Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid having tablet metadata in memory for queued compaction jobs #5188

Open
keith-turner opened this issue Dec 15, 2024 · 0 comments · May be fixed by #5247
Open

Avoid having tablet metadata in memory for queued compaction jobs #5188

keith-turner opened this issue Dec 15, 2024 · 0 comments · May be fixed by #5247
Assignees
@keith-turner
Labels
enhancement This issue describes a new feature, improvement, or optimization.
Milestone
4.0.0

Comments

@keith-turner
Copy link
Contributor

keith-turner commented Dec 15, 2024 •
edited
Loading

Is your feature request related to a problem? Please describe.

Currently when a compaction job starts executing it goes through the following process.

  1. Tablet group watcher finds a tablet that needs to compact
  2. The compaction jobs for the tablet and the tablet metadata for the tablet is placed on a priority queue
  3. Eventually a compaction job is pulled from the queue
  4. Then the tablet metadata is examined to see if its a candidate for compaction (sometimes the metadata is reread)
  5. Then a conditional mutation is submitted to try to reserve the compaction job

When lots of tablet have lots of files (this could happen when compactions are not running for some period of time), keeping lots of tablet metadata objects in memory could cause lots of memory pressure on the manager. This could lead to cascading failures where when the rest of the system is unhealthy it causes the manager to become unhealthy, leaving the manager unable to work through a temporary problem.

Describe the solution you'd like

It's probably possible to stop keeping the tablet metadata in memory when a compaction job is queued. This would make memory usage scale with the number of files in compaction jobs instead of the number of files in tablets, which is much better. The following change would also make compaction reservation more efficient as it would avoid reading tablet metadata in some cases and then submitting a conditional mutation.

  1. Tablet group watcher finds a tablet that needs to compact
  2. The compaction jobs for the tablet is placed on priority queue (the tablet metadata is no longer included)
  3. Eventually a compaction job is pulled from the queue
  4. The compaction job is included in a condition on a conditional mutation that does all checks to see if its safe to add the compaction job to the tablets metadata.

This process would use less memory overall and would streamline compaction reservation likely decreasing the time that it takes to atomically reserve a set of files for compaction for a tablet. This change could also simplify the code.
@keith-turner keith-turner added the enhancement This issue describes a new feature, improvement, or optimization. label Dec 15, 2024
@keith-turner keith-turner added this to the 4.0.0 milestone Dec 15, 2024
@keith-turner keith-turner mentioned this issue Dec 15, 2024
Merged
@cshannon cshannon mentioned this issue Dec 20, 2024
Open
@keith-turner keith-turner self-assigned this Jan 9, 2025
@keith-turner keith-turner mentioned this issue Jan 10, 2025
Open
keith-turner added a commit to keith-turner/accumulo that referenced this issue Jan 11, 2025
@keith-turner
Avoids using tablet metadata for compaction reservation
1b4c876 
This change fixes apache#5188.  Unfortunately it touches a lot of code because
of cascading dependencies in the code.  It would be difficult to break
this into a smaller commit.  These changes do reduce some of those
dependencies though.

There are two major advantages after this change.  First tablet metadata
is no longer kept in memory for queued compactions.  Second the tablet
metadata is no longer read during compaction reservation.  Before this
change the following would happen.

 1. TGW would find a tablet to compact and enqueue compaction
    jobs+tablet metadata.
 2. Eventually when a compactor requested a job it would yank job+tablet
    metadata off the queue.
 3. To reserve the compaction a lot of complex analysis was done in the
    coordinator and then a conditional mutation was submitted.  The
    conditional mutation would require all data involved in the complex
    analysis to be the same.
 4. If the conditional mutation failed then the coordinator would
    reread the tablet metadata and go back to step 3.

After this change the following happens in the code.

 1. TGW would find a tablet to compact and enqueue a new class called
    ResolvedCompactionJob.  This new class takes the compaction job and
    tablet metadata and computes all information needed for the compaction
    later.  The TabletMetadata object is no longer refrenced by this class
    after the constructor returns.  So this class will use much less memory
    on the queue for the case when tablet have lots of files.
 2. Eventually when a compactor requested a job it would yank a
    ResolvedCompactionJob off the queue.
 3. All of the complex analysis to determine if a compaction can start
    is now done in the conditional mutation instead of in the
    coordinator.  To enable this, new functionality was added to Ample
    including the new TabletMetadataCheck interface,
    TabletMetadataCheckIterator, and the CompactionReservationCheck class.
    With these changes its now super easy to write a conditional check that
    will do arbitrary analysis of TabletMetadata prior to committing a
    mutation.
 4. Since the analysis is done in the conditional mutation there is no
    longer a need to retry.  If the mutation fails then we know the
    compaction can not run.

The following were some supporting changes that had to be made.

 * Took methods for encoding KeyExtent as base64 from
   TabletManagementParameters and moved the KeyExtent because this was
   needed in the new TabletMetadataCheckIterator.
 * Move TabletMetadata out of the compaction queues, which made those
   more independent but was a big change.  The main change here is that
   instead of adding `TabletMetadata, List<CompactionJob>` to the
   compaction queue now `KeyExtent, List<CompactionJob>` is added.  This
   required changing the test for these classes and the code that interacts
   with them in CompactionCoordinator creating lots of diff.
 * Moved CompactionCoordinatorTest.testCanReserve() to
   CompactionReservationCheckTest.testCanReserve() and changed the code
   to work with the new CompactionReservationCheck class.  These are test
   of the complex logic that used to run in the coordinator and now runs in
   the tablet server as part of a conditional mutation.
 * Removed conditional checks from Ample related to compactions that
   were no longer used after these changes.  There was code for
   requiring a set of files not not be compacting.

The new TabletMetadataCheck functionality of ample could be used to
simplify other conditional mutation checks of tablet metadata.  It made
the compaction reservation code much simpler and easier to understand.
This code could be further improved if apache#5244 were implemented.
@keith-turner keith-turner linked a pull request Jan 11, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@keith-turner keith-turner

Labels
enhancement This issue describes a new feature, improvement, or optimization.
Projects
None yet
Milestone
4.0.0
Development

Successfully merging a pull request may close this issue.

Avoids holding tablet metadata for queued compactions keith-turner/accumulo
1 participant
@keith-turner