-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize processing of incoming blocks #529
Comments
heifner
added a commit
that referenced
this issue
Aug 13, 2024
heifner
added a commit
that referenced
this issue
Aug 13, 2024
heifner
added a commit
that referenced
this issue
Aug 13, 2024
…e add() indication of best head to know if block should be posted to the main thread for processing.
heifner
added a commit
that referenced
this issue
Aug 19, 2024
heifner
added a commit
that referenced
this issue
Aug 19, 2024
heifner
added a commit
that referenced
this issue
Aug 20, 2024
heifner
added a commit
that referenced
this issue
Aug 20, 2024
heifner
added a commit
that referenced
this issue
Aug 22, 2024
heifner
added a commit
that referenced
this issue
Aug 22, 2024
heifner
added a commit
that referenced
this issue
Sep 21, 2024
heifner
added a commit
that referenced
this issue
Sep 23, 2024
…for irreversible to make it clearer.
heifner
added a commit
that referenced
this issue
Sep 23, 2024
heifner
added a commit
that referenced
this issue
Sep 23, 2024
heifner
added a commit
that referenced
this issue
Sep 23, 2024
heifner
added a commit
that referenced
this issue
Sep 23, 2024
…' into GH-529-optimize-block-processing-part2
heifner
added a commit
that referenced
this issue
Sep 23, 2024
heifner
added a commit
that referenced
this issue
Sep 24, 2024
…-part2' into GH-529-optimize-block-processing-part3
heifner
added a commit
that referenced
this issue
Sep 25, 2024
heifner
added a commit
that referenced
this issue
Sep 25, 2024
… need to serialize those through the dispatcher strand. Check for LIB on processing block. Test failed because the dispatcher strand was way behind all other threads.
heifner
added a commit
that referenced
this issue
Sep 25, 2024
… was increased to make it more stable
heifner
added a commit
that referenced
this issue
Sep 25, 2024
heifner
added a commit
that referenced
this issue
Sep 25, 2024
…s nice on startup when there are many blocks in the forkdb.
heifner
added a commit
that referenced
this issue
Sep 26, 2024
heifner
added a commit
that referenced
this issue
Sep 26, 2024
heifner
added a commit
that referenced
this issue
Sep 26, 2024
…ing-part2 Optimize block processing - Part 2
heifner
added a commit
that referenced
this issue
Sep 26, 2024
…ing-part3 Optimize block processing - Part 3
heifner
added a commit
that referenced
this issue
Oct 1, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In Leap <= 5.0, blocks were only added to the fork database on the main thread. Therefore, blocks would be posted to the main thread for processing when they were not linkable. They were often linkable by the time the main thread got around to processing them as their previous block was often posted to the main thread for processing right before them.
In Spring, we add blocks to the fork database as soon as they are received; often way before they are applied. For example, during syncing with a
sync-fetch-span
of 1000, it is not uncommon for all 1000 blocks to be added to the fork database before any are applied on the main thread. Now, that all received blocks are immediately added to the fork database; if a block is unlinkable, there is no reason to post that block to the main thread for processing. There will be no blocks ahead of the posted block that would allow it to be unlinkable as all blocks are already in the fork database. Therefore, if a block is unlinkable when it is immediately placed in the fork database, that block can be rejected without any additional processing.Additionally, there is no need to post a block to the main thread for processing if the block is not the new best head of the fork database. Only blocks that can be applied to the previous head or that cause a fork switch need to be processed. All other blocks only need to be added to the fork database for future potential use in a fork switch.
Also, there is no need to actually post the block to the main thread for processing as it is readily available from the fork database. It is preferable to not post the block to the main thread as the new best head of the fork database can change from the time of posting of the block to the main thread to the time of execution of that task on the main thread. Therefore, we can simply signal that block processing from the fork database is potentially needed. The main thread can respond to that signal by evaluating what blocks need to be applied (or fork switched) from the fork database.
Note, that we expect many times for there to be no work required by the main thread when it receives the task to apply blocks from the fork database. Take for example, the 1000 block syncing from above, in that scenario the first task posted to the main thread can process all 1000 blocks as opposed to processing only one block. The other 1000 tasks posted to the main thread will see that there is nothing to do. See #284.
Another possible enhancement is to add an enum tag to the main thread post which would allow the post call to check if the back of the queue already contains the same enum tag and empty lambda. If so, the post call could no-op as the existing unprocessed posted task will suffice. There is no need to post duplicate signals to process blocks from the fork database as the first post task will process all available. Note that #284 can be implemented to process a subset of the blocks in the fork database and post an additional enum tag and empty lambda for future processing of the rest of the blocks.
The text was updated successfully, but these errors were encountered: