Proposed version 2.3.1 #5243

vlntb · 2025-01-13T21:10:02Z

High Level Overview of Change

This is a hotfix release that includes the following updates:

A fix for an erroneous high fee penalty that peers could incur for sending older transactions;
Updates to the fees charged for imposing a load on the server;
A fix to prevent the relaying of internal pseudo-transactions;
Improved logging.

Context of Change

Type of Change

Bug fix (non-breaking change which fixes an issue)
Release

API Impact

Public API: New feature (new methods and/or new fields)
Public API: Breaking change (in general, breaking changes should only impact the next api_version)
libxrpl change (any change that may affect libxrpl or dependents of libxrpl)
Peer protocol change (must be backward compatible or bump the peer protocol version)

codecov · 2025-01-13T21:32:07Z

Codecov Report

Attention: Patch coverage is 27.37430% with 130 lines in your changes missing coverage. Please review.

Project coverage is 77.9%. Comparing base (f64cf91) to head (59820e3).

Files with missing lines	Patch %	Lines
src/xrpld/overlay/detail/PeerImp.cpp	1.0%	98 Missing ⚠️
src/xrpld/overlay/detail/OverlayImpl.cpp	68.0%	8 Missing ⚠️
src/xrpld/overlay/detail/PeerImp.h	0.0%	7 Missing ⚠️
src/xrpld/app/ledger/detail/InboundLedger.cpp	0.0%	5 Missing ⚠️
...rc/xrpld/app/ledger/detail/InboundTransactions.cpp	0.0%	4 Missing ⚠️
src/xrpld/app/ledger/detail/LedgerMaster.cpp	0.0%	4 Missing ⚠️
src/libxrpl/resource/Charge.cpp	0.0%	2 Missing ⚠️
include/xrpl/resource/detail/Logic.h	92.9%	1 Missing ⚠️
src/xrpld/rpc/detail/ServerHandler.cpp	88.9%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           release   #5243     +/-   ##
=========================================
- Coverage     77.9%   77.9%   -0.1%     
=========================================
  Files          784     784             
  Lines        66681   66742     +61     
  Branches      8162    8184     +22     
=========================================
+ Hits         51950   51963     +13     
- Misses       14731   14779     +48

Files with missing lines	Coverage Δ
src/libxrpl/protocol/BuildInfo.cpp	`98.2% <ø> (ø)`
src/libxrpl/resource/Consumer.cpp	`89.8% <100.0%> (ø)`
src/xrpld/overlay/Overlay.h	`70.0% <ø> (ø)`
src/xrpld/overlay/Peer.h	`100.0% <ø> (ø)`
src/xrpld/overlay/detail/OverlayImpl.h	`40.0% <ø> (ø)`
src/xrpld/rpc/detail/RPCHelpers.cpp	`82.8% <100.0%> (ø)`
src/xrpld/rpc/handlers/GatewayBalances.cpp	`89.8% <100.0%> (ø)`
src/xrpld/rpc/handlers/LedgerHandler.cpp	`22.6% <100.0%> (ø)`
src/xrpld/rpc/handlers/PathFind.cpp	`48.3% <100.0%> (ø)`
src/xrpld/rpc/handlers/RipplePathFind.cpp	`42.5% <100.0%> (ø)`
... and 12 more

... and 4 files with indirect coverage changes

* Before: Pseudo-transactions received from a peer will fail the signature check, even if they were requested (using TMGetObjectByHash), because they have no signature. This causes the peer to be charge for an invalid signature. * After: Pseudo-transactions, are put into the global cache (TransactionMaster) only. If the transaction is part of a TMTransactions batch, the peer is charged the equivalent of one trivial request per transaction. If not, the peer is charged an unwanted data fee. These fees will not be a problem in the normal course of operations, but should dissuade peers from behaving badly by sending a bunch of junk.

* If reduce relay is enabled, it will queue up the tx id for peers that also have it enabled so they can ask for it later if they need it.

ximinez

Go ahead and squash the last two commits (version and release notes) into one, and change the commit message to follow the standard format: "Set version to 2.3.1"

src/test/overlay/tx_reduce_relay_test.cpp

bthomee · 2025-01-24T14:08:05Z

include/xrpl/resource/detail/Logic.h

+        // Only use these for logging in Logic::charge, and keep the values in
+        // descending order
+        static Charge const feeLogAsWarn(3000, "log as warn");
+        static Charge const feeLogAsInfo(1000, "log as info");
+        static Charge const feeLogAsDebug(100, "log as debug");


I would recommend defining these in Fees.h/cpp so the consequences of changing any of the fees are more apparent in terms of how they also will be logged.

I actually had the opposite thought - now that I see them here, there's no need to define these as Charges. They can just be straight up ints. (Or more specifically, Charge::value_type to ensure the types remain consistent.)
e.g.

static constexpr Charge::value_type feeLogAsWarn = 3000;

Even if we change the values in Fees.cpp in the future, it wouldn't make sense to change them so radically that these cutoffs should ever need to change. e.g. There should always be something that only charges 1. Even the things we changed, we made the values larger, so if anything, they'll log at a higher level, which is probably what we want.

I think a comment in Fees.cpp would be sufficient.

// See also Resource::Logic::charge for log level cutoff values

Now, all that said, if you still disagree, I don't object to moving them to Fees.cpp 😄

Edit: And if you notice there, I used constexpr, so you could add a quick static_assert that won't hurt anything.

static_assert(feeLogAsWarn > feeLogAsInfo && feeLogAsInfo > feeLogAsDebug && feeLogAsDebug > 10);

I'm not sure about this. Putting these definitions into Fees.h/cpp will encourage someone to use them as actual fees mistakenly. Keeping them close to the log-selection logic makes more sense to me.

ximinez · 2025-01-24T15:06:16Z

include/xrpl/resource/detail/Logic.h

+        // Only use these for logging in Logic::charge, and keep the values in
+        // descending order
+        static Charge const feeLogAsWarn(3000, "log as warn");
+        static Charge const feeLogAsInfo(1000, "log as info");
+        static Charge const feeLogAsDebug(100, "log as debug");


I actually had the opposite thought - now that I see them here, there's no need to define these as Charges. They can just be straight up ints. (Or more specifically, Charge::value_type to ensure the types remain consistent.)
e.g.

static constexpr Charge::value_type feeLogAsWarn = 3000;

Even if we change the values in Fees.cpp in the future, it wouldn't make sense to change them so radically that these cutoffs should ever need to change. e.g. There should always be something that only charges 1. Even the things we changed, we made the values larger, so if anything, they'll log at a higher level, which is probably what we want.

I think a comment in Fees.cpp would be sufficient.

// See also Resource::Logic::charge for log level cutoff values

Now, all that said, if you still disagree, I don't object to moving them to Fees.cpp 😄

Edit: And if you notice there, I used constexpr, so you could add a quick static_assert that won't hurt anything.

static_assert(feeLogAsWarn > feeLogAsInfo && feeLogAsInfo > feeLogAsDebug && feeLogAsDebug > 10);

bthomee · 2025-01-24T19:57:26Z

src/xrpld/overlay/detail/OverlayImpl.cpp

+    if (relay)
+    {
+        auto& txn = tx->get();
+        sm = std::make_shared<Message>(txn, protocol::mtTRANSACTION);
+    }
+
+    if (!relay && !app_.config().TX_REDUCE_RELAY_ENABLE)
+        return;


It looks like this can be simplified to:

if (!relay) { auto peers = getActivePeers(toSkip, 0, 0, 0) JLOG(journal_.trace()) << "not relaying tx, total peers " << peers.size(); for (auto const& p : peers) p->addTxQueue(hash); return; } auto& txn = tx->get(); sm = std::make_shared<Message>(txn, protocol::mtTRANSACTION);

And then the statements below no longer need to check whether relay is set.

Message allocation is more expensive than couple of additional boolean checks. The current approach should have performance advantage.

There should be no performance disadvantage to my proposal, since in the current code the message is allocated when relay is set, which is also the case in my proposal - the code returns early when relay is not set.

the relay initial is set in here:

bool relay = tx.has_value();

which guarantees that transaction is initialised.
If we remove the check, then later when we try to unbox transaction from optional

auto& txn = tx->get();

we may get a runtime error.

bthomee

LGTM for everything else.

src/xrpld/overlay/detail/PeerImp.cpp

bthomee · 2025-01-24T20:08:14Z

src/xrpld/overlay/detail/PeerImp.cpp

            break;
        case ListDisposition::invalid:
            // This shouldn't ever happen with a well-behaved peer
-            fee_ = Resource::feeInvalidSignature;
+            fee_.update(
+                Resource::feeInvalidSignature, "invalid list disposition");


Is invalid signature here appropriate? If not but we just want to charge the peer a lot, you can create a new const fee.

This error is returned from

// The returned PublicKey value is read from the manifest. Manifests do not // contain the default-constructed public keys std::pair<ListDisposition, std::optional<PublicKey>> ValidatorList::verify

from several places including cryptographic signature check.

I agree that we need to look into this and potentially add more granular error types here, because we may get this error for non-cryptographic reasons too, but I wouldn't suggest doing this as part of the hotfix.

Co-authored-by: Bart <[email protected]>

Bronek · 2025-01-24T21:11:30Z

src/xrpld/overlay/detail/PeerImp.h

+            assert(f >= fee);
+            fee = f;


reading how we call fee_.update in PeerImp.cpp I do not see anything to provide the guarantee that this assert wants. Should this be fee = std::max(f, fee); instead ?

... or even if (f < fee) return;

handler.onMessageBegin( header.message_type, m, header.payload_wire_size, header.uncompressed_size, header.algorithm != Algorithm::None); handler.onMessage(m); handler.onMessageEnd(header.message_type, m);

We always start from onMessageBegin, which sets the fee to the lowest level (feeTrivialPeer = 1).
We only process one message in this run.
This guarantees that fee update always goes from 1 to any other fee that is higher.

Bronek · 2025-01-24T21:17:46Z

RELEASENOTES.md

+
+### Bug Fixes and Performance Improvements
+
+- Change the charged fee for sending older transactions from feeInvalidSignature to feeUnwantedData. [#5243](https://github.com/XRPLF/rippled/pull/5243)


Change this line; preferably into multiple items (change fees overall, do not charge repeated fees on batch transactions, improve logging of fees, not sure how to describe changes to relay)

Bronek · 2025-01-24T21:19:33Z

src/xrpld/overlay/detail/PeerImp.cpp

+            if (tx->getStatus() == NEW)
+            {
+                JLOG(p_journal_.debug())
+                    << "Cacheing " << (batch ? "batch" : "unsolicited")


src/xrpld/overlay/detail/PeerImp.cpp

src/libxrpl/protocol/BuildInfo.cpp

Co-authored-by: Ed Hennis <[email protected]>

src/xrpld/overlay/detail/OverlayImpl.cpp

…ippled into fix/signaturefee-to-unwanteddata

Co-authored-by: Ed Hennis <[email protected]>

bthomee · 2025-01-25T21:33:55Z

src/xrpld/overlay/detail/PeerImp.cpp

+    // Something new: Dynamic fee
+    fee_.fee = Resource::Charge(m->transactions_size(), "transactions");


Given that the existing code does not have a corresponding fee for one or more transactions, what is the practical effect of this change?

The handleTransaction function is called for each transaction and already applies charges. Does charging this dynamic fee here cause double-charging or does it serve to fill a gap that currently exists?

bthomee · 2025-01-25T21:49:16Z

src/xrpld/overlay/detail/PeerImp.cpp

+                JLOG(p_journal_.debug())
+                    << "Caching " << (batch ? "batch" : "unsolicited")
+                    << " pseudo-transaction tx " << tx->getID();


This log message doesn't seem to match what is happening here. There's canonicalization and possibly relaying going on, while the log message suggests that both batch and non-batch (unsolicited) tx are being cached.

I'd also recommend just having three separate and appropriately phrased log messages, depending on the situation below (skipped, relayed, charged).

Also, to confirm:

One of the fixes introduced in this hotfix was to not relay pseudo-tx, but isn't that what actually can happen here anyway?

If a pseudo-tx is non-batch it is unwanted - is there any point to canonicalize and possibly relay it then, before charging the peer for useless data? If possible, moving the if (!batch) check up and then returning early would seem sensible.

vlntb added 2 commits January 13, 2025 21:08

fix fee from feeInvalidSignature to feeUnwantedData

568c567

update version to 2.3.1

bd6ef12

alloynetworks mentioned this pull request Jan 14, 2025

Older Transaction processing message Xahau/xahaud#422

Open

update release notes

0176e60

vlntb changed the title ~~fix fee from feeInvalidSignature to feeUnwantedData~~ Proposed version 2.3.1 Jan 14, 2025

vlntb requested a review from ximinez January 14, 2025 12:59

vlntb marked this pull request as ready for review January 14, 2025 13:00

ximinez added 4 commits January 14, 2025 20:00

Tell the overlay about pseudo transactions, but don't relay them

6ec144e

* If reduce relay is enabled, it will queue up the tx id for peers that also have it enabled so they can ask for it later if they need it.

Log higher resource charges with higher log levels

4c0608d

Include context on (some) resource charges

cd9d55e

ximinez requested changes Jan 15, 2025

View reviewed changes

ximinez and others added 3 commits January 15, 2025 12:01

Add context to all Peer charges

fb04007

feeMediumBurdenPeer and feeInvalidSignature context details

73e8ae3

merge Eds changes

1141201

vlntb marked this pull request as draft January 16, 2025 18:19

vlntb added 6 commits January 17, 2025 18:02

stop relaying pseudo-tx

fcf8eb2

increased dropThreshold and feeHighBurdenPeer

4f58fef

increased the fees

238aeff

fixing formatting

6ee3339

fixing unit test

03d79e5

missing header

35b0690

ximinez requested changes Jan 21, 2025

View reviewed changes

src/test/overlay/tx_reduce_relay_test.cpp Outdated Show resolved Hide resolved

vlntb and others added 6 commits January 22, 2025 17:12

fix fee init - missing ctor

6a5025b

code refactoring

423a7f3

addressing review comment

873e855

update levelization

d62c216

fixing formatting

9902829

Limit the size of outgoing mtLEDGER_DATA / TMLedgerData messages

66feee4

renaming

16b8007

bthomee reviewed Jan 24, 2025

View reviewed changes

vlntb added 2 commits January 24, 2025 15:31

removing spaces

7b63b8d

adding spaceship operator

2579458

ximinez requested changes Jan 24, 2025

View reviewed changes

vlntb and others added 3 commits January 24, 2025 18:16

unit test update

5d1a05d

revert

b447691

fee logging

740d18c

vlntb requested review from bthomee and ximinez January 24, 2025 19:21

bthomee reviewed Jan 24, 2025

View reviewed changes

Update src/xrpld/overlay/detail/PeerImp.cpp

b5947ce

Co-authored-by: Bart <[email protected]>

Bronek reviewed Jan 24, 2025

View reviewed changes

ximinez changed the base branch from master to release January 24, 2025 21:36

ximinez requested changes Jan 24, 2025

View reviewed changes

src/xrpld/overlay/detail/PeerImp.cpp Outdated Show resolved Hide resolved

ximinez requested changes Jan 24, 2025

View reviewed changes

src/libxrpl/protocol/BuildInfo.cpp Outdated Show resolved Hide resolved

vlntb and others added 2 commits January 24, 2025 21:48

Update src/xrpld/overlay/detail/PeerImp.cpp

c46c002

Co-authored-by: Ed Hennis <[email protected]>

Update src/libxrpl/protocol/BuildInfo.cpp

cf0c2a1

Co-authored-by: Ed Hennis <[email protected]>

ximinez requested changes Jan 24, 2025

View reviewed changes

src/xrpld/overlay/detail/OverlayImpl.cpp Outdated Show resolved Hide resolved

vlntb and others added 4 commits January 24, 2025 21:50

release notes and typo fix

900d383

Merge branch 'fix/signaturefee-to-unwanteddata' of github.com:vlntb/r…

0eeeecd

…ippled into fix/signaturefee-to-unwanteddata

Update src/xrpld/overlay/detail/OverlayImpl.cpp

13974f4

Co-authored-by: Ed Hennis <[email protected]>

finished refactoring relay

59820e3

vlntb requested review from bthomee, ximinez and Bronek January 25, 2025 07:03

bthomee reviewed Jan 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposed version 2.3.1 #5243

Proposed version 2.3.1 #5243

vlntb commented Jan 13, 2025 •

edited

Loading

codecov bot commented Jan 13, 2025 •

edited

Loading

ximinez left a comment

bthomee Jan 24, 2025

ximinez Jan 24, 2025 •

edited

Loading

vlntb Jan 24, 2025

ximinez Jan 24, 2025 •

edited

Loading

bthomee Jan 24, 2025 •

edited

Loading

vlntb Jan 24, 2025

bthomee Jan 24, 2025

vlntb Jan 24, 2025 •

edited

Loading

bthomee left a comment

bthomee Jan 24, 2025

vlntb Jan 24, 2025

Bronek Jan 24, 2025

Bronek Jan 24, 2025

vlntb Jan 24, 2025 •

edited

Loading

Bronek Jan 24, 2025

Bronek Jan 24, 2025

bthomee Jan 25, 2025

bthomee Jan 25, 2025


		### Bug Fixes and Performance Improvements

		- Change the charged fee for sending older transactions from feeInvalidSignature to feeUnwantedData. [#5243](https://github.com/XRPLF/rippled/pull/5243)

		// Something new: Dynamic fee
		fee_.fee = Resource::Charge(m->transactions_size(), "transactions");

Proposed version 2.3.1 #5243

Are you sure you want to change the base?

Proposed version 2.3.1 #5243

Conversation

vlntb commented Jan 13, 2025 • edited Loading

High Level Overview of Change

Context of Change

Type of Change

API Impact

codecov bot commented Jan 13, 2025 • edited Loading

Codecov Report

ximinez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ximinez Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ximinez Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

bthomee Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vlntb Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

bthomee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vlntb Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vlntb commented Jan 13, 2025 •

edited

Loading

codecov bot commented Jan 13, 2025 •

edited

Loading

ximinez Jan 24, 2025 •

edited

Loading

ximinez Jan 24, 2025 •

edited

Loading

bthomee Jan 24, 2025 •

edited

Loading

vlntb Jan 24, 2025 •

edited

Loading

vlntb Jan 24, 2025 •

edited

Loading