Improve transaction relay logic

[ximinez]

High Level Overview of Change

This PR, if merged, will improve transaction relay logic around a few edge cases.

(I'll write a single commit message later, but before this PR is squashed and merged.)

Context of Change

A few months ago, while examining some of the issues around the 2.0.0 release, and auditing transaction relay code, I identified a few areas with potential for improvement.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Tests (you added tests for code that already exists, or your new feature included in this PR)

Before / After

This PR is divided into four mostly independent changes.

1. "Decrease shouldRelay limit to 30s." Pretty self-explanatory. Currently, the limit is 5 minutes, by which point the HashRouter entry could have expired, making this transaction look brand new (and thus causing it to be relayed back to peers which have sent it to us recently).
2. "Give a transaction more chances to be retried." Will put a transaction into LedgerMaster's held transactions if the transaction gets a ter, tel, or tef result. Old behavior was just ter.
   • Additionally, to prevent a transaction from being repeatedly held indefinitely, it must meet some extra conditions. (Documented in a comment in the code.)
3. "Pop all transactions with sequential sequences, or tickets." When a transaction is processed successfully, currently, one held transaction for the same account (if any) will be popped out of the held transactions list, and queued up for the next transaction batch. This change pops all transactions for the account, but only if they have sequential sequences (for non-ticket transactions) or use a ticket. This issue was identified from interactions with @mtrippled's Apply transaction batches in periodic intervals. #4504, which was merged, but unfortunately reverted later by Revert "Apply transaction batches in periodic intervals (#4504)" #4852. When the batches were spaced out, it could potentially take a very long time for a large number of held transactions for an account to get processed through. However, whether batched or not, this change will help get held transactions cleared out, particularly if a missing earlier transaction is what held them up.
4. "Process held transactions through existing NetworkOPs batching." In the current processing, at the end of each consensus round, all held transactions are directly applied to the open ledger, then the held list is reset. This bypasses all of the logic in NetworkOPs::apply which, among other things, broadcasts successful transactions to peers. This means that the transaction may not get broadcast to peers for a really long time (5 minutes in the current implementation, or 30 seconds with this first commit). If the node is a bottleneck (either due to network configuration, or because the transaction was submitted locally), the transaction may not be seen by any other nodes or validators before it expires or causes other problems.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 81.66667% with 22 lines in your changes missing coverage. Please review.

Project coverage is 77.9%. Comparing base (040cd23) to head (a35aab0).

Files with missing lines	Patch %	Lines
src/xrpld/app/misc/NetworkOPs.cpp	82.6%	15 Missing ⚠️
src/xrpld/app/misc/HashRouter.cpp	56.2%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #4985     +/-   ##
=========================================
- Coverage     77.9%   77.9%   -0.0%     
=========================================
  Files          783     783             
  Lines        66707   66788     +81     
  Branches      8118    8129     +11     
=========================================
+ Hits         51954   52014     +60     
- Misses       14753   14774     +21     

Files with missing lines	Coverage Δ
src/xrpld/app/ledger/LocalTxs.h	100.0% <ø> (ø)
src/xrpld/app/ledger/detail/LedgerMaster.cpp	43.9% <100.0%> (-<0.1%)	⬇️
src/xrpld/app/ledger/detail/LocalTxs.cpp	100.0% <100.0%> (ø)
src/xrpld/app/main/Application.cpp	68.8% <100.0%> (-0.1%)	⬇️
src/xrpld/app/misc/CanonicalTXSet.cpp	100.0% <100.0%> (ø)
src/xrpld/app/misc/HashRouter.h	100.0% <100.0%> (ø)
src/xrpld/app/misc/NetworkOPs.h	100.0% <ø> (ø)
src/xrpld/app/misc/Transaction.h	100.0% <ø> (ø)
src/xrpld/app/misc/HashRouter.cpp	89.9% <56.2%> (-10.1%)	⬇️
src/xrpld/app/misc/NetworkOPs.cpp	70.5% <82.6%> (+0.4%)	⬆️

... and 2 files with indirect coverage changes

[scottschurr]

Very nice! I really appreciate the way the commits were divided up. It made the code review much easier.

I mostly left complementary comments. But there's one bool that I suspect needs to be changed to a std::atomic<bool>. See what you think...

[scottschurr]

Nice! This takes full advantage of the "unusual" sort order of SeqProxy, that all sequence numbers sort in front of all tickets.

[scottschurr]

I was initially worried that this while loop might submit_held() a boatload of transactions. But the TxQ defaults maximumTxnPerAccount to 10. So the largest number of times this loop could run (ordinarily) would be 10. That seems reasonable, and a good way to clear out an account that has a lot of transactions queued.

This new characteristic may be worth pointing out to the performance folks, in case they want to stress it.

[ximinez]

To clarify, this loop doesn't do anything with the TxQ. This is reading the list of held transactions in LedgerMaster. So, AFAIK, there is no limit. However, the only way for a tx to get into that list is in this same function (see calls to addHeldTransaction), and only transactions received directly from RPC or from peers get to this function. That said, a malicious client or peer could probably get a bunch of transactions held (this is true without these changes). Thus, it might make sense to add a limit to mitigate some kinds of attacks.

Also, submit_held is a std::vector. Anything added to it will be processed in the next batch of transactions.

[scottschurr]

This is a great way to minimize processing while the lock is held. Good spotting!

[ximinez]

Thanks!

[scottschurr]

Splitting the validation and canonicalization part of processTransaction into this other method was a good idea.

[ximinez]

Thanks!

[scottschurr]

I looked into this call to Transaction::getApplying(). The bool being read is neither protected by a lock nor atomic. I think we need to change the Transaction::mApplying member variable into a std::atomic<bool>, since the bool is being accessed across threads.

This problem was present before your change. But please fix it while we're thinking about it.

[ximinez]

I think this was done intentionally because it avoids the overhead of taking a lock, and the consequences of a race condition are nearly-zero.

1. Most significantly, almost all accesses of the *Applying functions are under NetworkOPsImp's lock. The exceptions are
   1. The new processTransactionSet function.
   2. The popAcctTransaction loop we were just talking about.
2. If there is a race condition, and getApplying returns false when it should be true, the transaction will be processed again. Not that big a deal if it's a rare one-off. Most of the time, it'll get tefALREADY or tefPAST_SEQ.
3. On the flip side, if it returns true, when it should be false, then the transaction must have been attempted recently, so no big deal if it doesn't immediately get tried right away.
4. If there's a race between setApplying and clearApplying, and the flag ends up set, then a batch is about to try to process the transaction and will call clearApplying later. If it ends up cleared, then it might get attempted again later as is the case with item 2.

Alllll that said, I don't think it would hurt to rewrite those two lock exceptions I pointed out to ensure that the lock is taken for the function calls.

[vlntb]

If this optimistic locking approach doesn't create any harmful side effects, as you suggest @ximinez , it makes sense to keep the current approach. But let's put your explanation in a comment in the code to preserve it.

[scottschurr]

Similar to above, these 5 lines are not hit by the unit tests. 🤷

[ximinez]

It should be impossible to get to this line. Note the block above explains that I don't think the check is necessary, and has an assert on the same validity at the end.

    // NOTE eahennis - I think this check is redundant,
    // but I'm not 100% sure yet.
    // If so, only cost is looking up HashRouter flags.
    auto const view = m_ledgerMaster.getCurrentLedger();
    auto const [validity, reason] = checkValidity(
        app_.getHashRouter(),
        *transaction->getSTransaction(),
        view->rules(),
        app_.config());
    assert(validity == Validity::Valid);

[scottschurr]

Interesting. According to my local code coverage these four lines are never hit. I know there are a few places in the unit tests that produce corrupted signatures. They must be handled elsewhere. Just noticing, no need to address this in this pull request.

[ximinez]

Yeah, this one isn't documented as explicitly, but the idea is that it should also be impossible to get to this point with a bad signature.

[scottschurr]

Does it make sense to reserve space in mTransactions? Consider

mTransactions.reserve(mTransactions.size() + transactions.size());

[ximinez]

Yes it does!

[vlntb]

The change was thoroughly reviewed by former colleagues. I added a couple of questions. Once all the questions are answered, I will be happy to approve it.

[vlntb]

It would make sense to have RelayTime defined in the config. The 30s is not a formally proven optimal value. Once used in the liveness environment, we might want to adjust it.
The same can be said about the HoldTime.

[ximinez]

The HoldTime has been 300s for over 11 years, so it's a pretty safe bet that's a good value. However, I see your point about being able to tweak RelayTime.

I added a commit to add configuration options for these two values. I'm leaving them undocumented for now, because I don't want operators just randomly changing them for no particular reason.

[vlntb]

[nit]: It would drop lock parameter name in lambda. This will clarify the intent of the lock being required by the function signature, and we didn't forget to acquire the lock inside the lambda.

[ximinez]

Yeah, good idea. I also changed the function signature to require the lock to be const in the callback.

[vlntb]

If this optimistic locking approach doesn't create any harmful side effects, as you suggest @ximinez , it makes sense to keep the current approach. But let's put your explanation in a comment in the code to preserve it.

[vlntb]

If we only pop a maximum of ten transactions, does this mean that over time, we will have a slow memory leak here of transactions that will never be used but will stay in the CanonicalTXSet indefinitely?

[ximinez]

No, because the mHeldTransactions list that this function pulls from is also fully cleared out and retried all at once in LedgerMaster::applyHeldTransactions(). applyHeldTransactions is called during consensus when a new closed ledger is created from the open ledger to start the process.

There are restrictions on how many times a transaction can be put into mHeldTransactions so that it can't cycle in and out indefinitely. Those are described at one of the call sites of addHeldTransaction()

[vlntb]

All raised questions addressed.