Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize processing of incoming blocks #529

Closed
heifner opened this issue Aug 13, 2024 · 0 comments · Fixed by #619, #688 or #802
Closed

Optimize processing of incoming blocks #529

heifner opened this issue Aug 13, 2024 · 0 comments · Fixed by #619, #688 or #802
Assignees
Labels
👍 lgtm OCI Work exclusive to OCI team

Comments

@heifner
Copy link
Member

heifner commented Aug 13, 2024

In Leap <= 5.0, blocks were only added to the fork database on the main thread. Therefore, blocks would be posted to the main thread for processing when they were not linkable. They were often linkable by the time the main thread got around to processing them as their previous block was often posted to the main thread for processing right before them.

In Spring, we add blocks to the fork database as soon as they are received; often way before they are applied. For example, during syncing with a sync-fetch-span of 1000, it is not uncommon for all 1000 blocks to be added to the fork database before any are applied on the main thread. Now, that all received blocks are immediately added to the fork database; if a block is unlinkable, there is no reason to post that block to the main thread for processing. There will be no blocks ahead of the posted block that would allow it to be unlinkable as all blocks are already in the fork database. Therefore, if a block is unlinkable when it is immediately placed in the fork database, that block can be rejected without any additional processing.

Additionally, there is no need to post a block to the main thread for processing if the block is not the new best head of the fork database. Only blocks that can be applied to the previous head or that cause a fork switch need to be processed. All other blocks only need to be added to the fork database for future potential use in a fork switch.

Also, there is no need to actually post the block to the main thread for processing as it is readily available from the fork database. It is preferable to not post the block to the main thread as the new best head of the fork database can change from the time of posting of the block to the main thread to the time of execution of that task on the main thread. Therefore, we can simply signal that block processing from the fork database is potentially needed. The main thread can respond to that signal by evaluating what blocks need to be applied (or fork switched) from the fork database.

Note, that we expect many times for there to be no work required by the main thread when it receives the task to apply blocks from the fork database. Take for example, the 1000 block syncing from above, in that scenario the first task posted to the main thread can process all 1000 blocks as opposed to processing only one block. The other 1000 tasks posted to the main thread will see that there is nothing to do. See #284.

Another possible enhancement is to add an enum tag to the main thread post which would allow the post call to check if the back of the queue already contains the same enum tag and empty lambda. If so, the post call could no-op as the existing unprocessed posted task will suffice. There is no need to post duplicate signals to process blocks from the fork database as the first post task will process all available. Note that #284 can be implemented to process a subset of the blocks in the fork database and post an additional enum tag and empty lambda for future processing of the rest of the blocks.

@enf-ci-bot enf-ci-bot moved this to Todo in Team Backlog Aug 13, 2024
@arhag arhag added 👍 lgtm and removed triage labels Aug 13, 2024
@arhag arhag added this to the Spring v1.0.0-rc2 milestone Aug 13, 2024
heifner added a commit that referenced this issue Aug 13, 2024
…e add() indication of best head to know if block should be posted to the main thread for processing.
heifner added a commit that referenced this issue Aug 13, 2024
@heifner heifner self-assigned this Aug 14, 2024
@heifner heifner added the OCI Work exclusive to OCI team label Aug 14, 2024
@heifner heifner moved this from Todo to In Progress in Team Backlog Aug 14, 2024
heifner added a commit that referenced this issue Aug 19, 2024
heifner added a commit that referenced this issue Aug 19, 2024
heifner added a commit that referenced this issue Aug 19, 2024
@arhag arhag removed this from the Spring v1.0.0 Cusp milestone Aug 20, 2024
@BenjaminGormanPMP BenjaminGormanPMP added this to the Spring v1.1.0-rc1 milestone Aug 20, 2024
heifner added a commit that referenced this issue Aug 22, 2024
heifner added a commit that referenced this issue Aug 22, 2024
heifner added a commit that referenced this issue Sep 23, 2024
heifner added a commit that referenced this issue Sep 23, 2024
heifner added a commit that referenced this issue Sep 23, 2024
heifner added a commit that referenced this issue Sep 24, 2024
heifner added a commit that referenced this issue Sep 24, 2024
heifner added a commit that referenced this issue Sep 25, 2024
heifner added a commit that referenced this issue Sep 25, 2024
heifner added a commit that referenced this issue Sep 25, 2024
heifner added a commit that referenced this issue Sep 25, 2024
heifner added a commit that referenced this issue Sep 25, 2024
… need to serialize those through the dispatcher strand. Check for LIB on processing block. Test failed because the dispatcher strand was way behind all other threads.
heifner added a commit that referenced this issue Sep 25, 2024
heifner added a commit that referenced this issue Sep 25, 2024
heifner added a commit that referenced this issue Sep 25, 2024
…s nice on startup when there are many blocks in the forkdb.
heifner added a commit that referenced this issue Sep 26, 2024
heifner added a commit that referenced this issue Sep 26, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in Team Backlog Sep 26, 2024
heifner added a commit that referenced this issue Sep 26, 2024
…ing-part2

Optimize block processing - Part 2
heifner added a commit that referenced this issue Sep 26, 2024
…ing-part3

Optimize block processing - Part 3
heifner added a commit that referenced this issue Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
👍 lgtm OCI Work exclusive to OCI team
Projects
Status: Done
4 participants