IF: Process votes in a dedicated thread pool #24

heifner · 2024-04-11T18:54:26Z

Process vote_messages in their own dedicated thread pool. The size of the thread pool is configured by a new chain_plugin config option vote-threads. Vote propagation is disabled by default. Vote threads should be enabled on BPs and on canary nodes between BPs. Votes are required to flow from finalizer nodes to all BPs for finality to advance. vote-threads if not specified is set to 4 on BP nodes.

  --vote-threads arg                    Number of worker threads in vote
                                        processor thread pool. If set to 0,
                                        voting disabled, votes are not
                                        propagatged on P2P network. Defaults to
                                        4 on producer nodes.

Report 'Produced block' and 'Received block' known LIB instead of fork db root in Savanna because these log statements used to be called after log_irreversible, but are now called before log_irreversible. To avoid confusion under Savanna, report what the known LIB is instead of what is currently root of the fork db. In legacy mode these will still continue to report the root before the call to log_irreversible; therefore under legacy the updated LIB will be reported one block later.
vote_messages are processed first-come-first-serve.

Resolves #3
Resolves #4

…bling vote processing

…t easier to test.

…or loop instead.

…hake

…tion>

…at time of C production it will not get a block until A turn. Need to wait at least 7 seconds. Used 10 here to provide a bit of buffer.

…ncrease a bit.

heifner · 2024-04-12T13:32:40Z

tests/nodeos_snapshot_forked_test.py

@@ -144,7 +144,7 @@ def getSnapshotsCount(nodeId):
    while nonProdNode.verifyAlive() and count > 0:
        # wait on prodNode 0 since it will continue to advance, since defproducera and defproducerb are its producers
        Print("Wait for next block")
-        assert prodAB.waitForNextBlock(timeout=6), "Production node AB should continue to advance, even after bridge node is killed"
+        assert prodAB.waitForNextBlock(timeout=10), "Production node AB should continue to advance, even after bridge node is killed"


6 seconds is not enough time here. There are three producers A,B,C. If this happens to hit on C's turn to produce then it needs to wait at least 7 seconds for a block to be produced. Bumped up to 10 to give a bit of breathing room.

heifner · 2024-04-12T13:33:02Z

unittests/api_tests.cpp

@@ -1226,7 +1226,7 @@ BOOST_AUTO_TEST_CASE(checktime_pause_block_deadline_not_extended_while_loading_t
   // WASM load times on my machine was 35ms.
   // Since checktime only kicks in after WASM is loaded this needs to be large enough to load the WASM, but should be
   // considerably lower than the 150ms max_transaction_time
-   BOOST_CHECK_MESSAGE( dur < 50'000, "elapsed " << dur << "us" );
+   BOOST_CHECK_MESSAGE( dur < 65'000, "elapsed " << dur << "us" );


Faild on an asan run. Took 55ms.

… used to be reported after the call to log_irreversible, they are called before log_irreversible. To avoid confusion by users report what is the known LIB.

…S_ASSERT if configured as a producer with vote processing disabled.

libraries/chain/include/eosio/chain/vote_processor.hpp

libraries/chain/controller.cpp

libraries/chain/include/eosio/chain/vote_processor.hpp

ericpassmore · 2024-04-17T17:10:10Z

Note:start
group: IF
category: INTERNALS
summary: Process votes in a dedicated thread pool.
Note:end

…t-serve basis.

greg7mdp

I like it, I think it is simpler and easier to feel confident that it is correct.

libraries/chain/controller.cpp

plugins/net_plugin/net_plugin.cpp

libraries/chain/include/eosio/chain/vote_processor.hpp

tests/TestHarness/Cluster.py

libraries/chain/include/eosio/chain/vote_processor.hpp

…processed

libraries/chain/include/eosio/chain/vote_processor.hpp

heifner added 6 commits April 10, 2024 18:57

GH-3 Move vote processing off net threads into a dedicated thread pool

7fc69ac

GH-3 Track num_messages per connection

c3a9c95

GH-3 Make vote processor thread pool size configurable including disa…

6b0c4c4

…bling vote processing

GH-3 Fix issue with not re-locking after unlock

fba5561

GH-3 Shutdown nodeos on vote thread pool exception

1dcf7c1

GH-3 Add unittest for vote_processor. Modify vote_processor to make i…

295f7d4

…t easier to test.

heifner added the OCI Work exclusive to OCI team label Apr 11, 2024

heifner requested review from linh2931 and greg7mdp April 11, 2024 18:54

This was linked to issues Apr 11, 2024

Guarantee net_plugin processes votes after it has has processed the block the vote was for #3

Closed

Add option to disable vote processing and relaying #4

Closed

heifner added 11 commits April 11, 2024 14:56

GH-3 std::latch not available on all platforms; use std::atomic and f…

18c2fca

…or loop instead.

GH-3 ci/cd is slow, allow more time.

d5cef78

GH-3 Make test more robust by checking that node has received a hands…

da0106b

…hake

GH-3 Optimization: Don't emit duplicate votes

bf545e3

GH-3 Fix: num_messages.clear() called before decrement

d1fe636

GH-3 Avoid vote_message copies by using shared_ptr

f3ff3e8

GH-3 Avoid vote_message copies by using shared_ptr

ae02196

GH-3 Fix tests now that duplicates are not signaled

a6e4de8

GH-3 Fix use of packed_transaction_ptr over shared_ptr<packed_transac…

382f648

…tion>

GH-3 There are three producers A,B,C if the test happens to hit this …

508b584

…at time of C production it will not get a block until A turn. Need to wait at least 7 seconds. Used 10 here to provide a bit of buffer.

GH-3 Test failed on asan run because it took 55ms to load the WASM. I…

4d70ab3

…ncrease a bit.

heifner commented Apr 12, 2024

View reviewed changes

heifner added 6 commits April 12, 2024 09:35

Merge remote-tracking branch 'spring/savanna' into GH-3-process-votes

23b5739

GH-3 Report known lib instead of fork db root as these log statements…

1641a07

… used to be reported after the call to log_irreversible, they are called before log_irreversible. To avoid confusion by users report what is the known LIB.

GH-3 Change default --vote-threads to 0 to be disabled by default. EO…

4b4eca6

…S_ASSERT if configured as a producer with vote processing disabled.

GH-3 --vote-threads needed for all producers

895cef7

Merge remote-tracking branch 'spring/savanna' into GH-3-process-votes

796dc5b

GH-12 Add vote threads needed for unittests

c3bdd8a

heifner added 2 commits April 16, 2024 12:04

GH-3 Add better descriptions

027d278

GH-3 Default vote-threads to 4 when a block producer

8f26a50

greg7mdp reviewed Apr 16, 2024

View reviewed changes

GH-3 Simplify by adding a vote_signal_t type

6cc6820

greg7mdp reviewed Apr 16, 2024

View reviewed changes

libraries/chain/include/eosio/chain/vote_processor.hpp Outdated Show resolved Hide resolved

libraries/chain/include/eosio/chain/vote_processor.hpp Outdated Show resolved Hide resolved

heifner added 2 commits April 16, 2024 14:02

GH-3 Use chain pluging accept_votes to init p2p_accept_votes

166a580

GH-3 Add vote-threads to all nodes so bridge nodes process votes

83434d1

greg7mdp reviewed Apr 16, 2024

View reviewed changes

libraries/chain/include/eosio/chain/vote_processor.hpp Outdated Show resolved Hide resolved

heifner added 2 commits April 17, 2024 14:10

GH-3 Simplify vote_processor by processing votes on a first-come-firs…

042ce30

…t-serve basis.

GH-3 Fix use of iterator after erase

d80def5

heifner requested review from spoonincode, greg7mdp and linh2931 April 17, 2024 19:54

greg7mdp requested changes Apr 18, 2024

View reviewed changes

linh2931 reviewed Apr 18, 2024

View reviewed changes

heifner added 7 commits April 18, 2024 14:55

GH-3 Remove unneeded if

317dd61

GH-3 Move emit to controller.hpp and use in vote_processor

955a72a

GH-3 Use same value as default for producer

4a759b3

GH-3 Use unordered_map

51503c4

GH-3 Add better log message

b694aad

GH-3 Do not clear num_messages if there are votes in the index to be …

344b778

…processed

GH-3 More descriptive emit logs

ecc8cd4

greg7mdp approved these changes Apr 18, 2024

View reviewed changes

libraries/chain/include/eosio/chain/vote_processor.hpp Outdated Show resolved Hide resolved

GH-3 Fix spelling

8d3a838

heifner requested a review from linh2931 April 18, 2024 21:04

greg7mdp approved these changes Apr 18, 2024

View reviewed changes

linh2931 approved these changes Apr 19, 2024

View reviewed changes

heifner merged commit 5e5c176 into savanna Apr 19, 2024
36 checks passed

heifner deleted the GH-3-process-votes branch April 19, 2024 11:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IF: Process votes in a dedicated thread pool #24

IF: Process votes in a dedicated thread pool #24

heifner commented Apr 11, 2024 •

edited

Loading

heifner Apr 12, 2024

heifner Apr 12, 2024

ericpassmore commented Apr 17, 2024 •

edited

Loading

greg7mdp left a comment

IF: Process votes in a dedicated thread pool #24

IF: Process votes in a dedicated thread pool #24

Conversation

heifner commented Apr 11, 2024 • edited Loading

heifner Apr 12, 2024

Choose a reason for hiding this comment

heifner Apr 12, 2024

Choose a reason for hiding this comment

ericpassmore commented Apr 17, 2024 • edited Loading

greg7mdp left a comment

Choose a reason for hiding this comment

heifner commented Apr 11, 2024 •

edited

Loading

ericpassmore commented Apr 17, 2024 •

edited

Loading