Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed output batching #1615

Open
wants to merge 54 commits into
base: dev_2-0
Choose a base branch
from

Conversation

Oscar-Pepper
Copy link
Contributor

@Oscar-Pepper Oscar-Pepper commented Jan 7, 2025

on top of #1613

this PR introduces the Batcher as a new task in the Scanner that solves the following issues:

  • once a ScanWorker receives a task of X blocks, it must scan this range even if there is an unusually large amounts of outputs as it does not (and should not) have access to the wallet to update its now-smaller scan range for other workers to pick-up the rest
  • due to this, an arbitrarily large amount of memory might be needed to scan some ranges i.e. in the sandblast regions
  • when scanning linearly, the workers are both given scan tasks (which contain the optional seam blocks [or adjacent blocks to the scan range]) so the workers will not have access to these for continuity checks between the "seams" of scan ranges as the first worker will not have finished and therefore added the seam block to the wallet

the Batcher solves these issues by:

  • receiving the initial scan task and filling it with the compact blocks needed for scanning that range
  • when the outputs count exceeds a maximum, splits the scan task into two and sends the full batch to a ScanWorker for scanning
  • as its the only part of the sync engine that fetches compact blocks, it can manage the seam blocks by sending the highest block of the full batch to the new batch etc.
  • also saves the first and last blocks of the previous scan task so it can manage seam blocks when scanning linearly in all cases for full continuity check coverage

@Oscar-Pepper Oscar-Pepper marked this pull request as ready for review January 7, 2025 15:26
@Oscar-Pepper Oscar-Pepper requested a review from zancas January 7, 2025 15:26
This was referenced Jan 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant