-
Notifications
You must be signed in to change notification settings - Fork 448
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add new plugin for conditional update that makes mutation available #5244
Labels
enhancement
This issue describes a new feature, improvement, or optimization.
Milestone
Comments
keith-turner
added
the
enhancement
This issue describes a new feature, improvement, or optimization.
label
Jan 10, 2025
Opened this issue based on problems I am seeing while working on #5188. Once I have a PR up for that can point to more specific example in that code that illustrate the problem. |
keith-turner
changed the title
Add new pluging for conditional update that make mutation available
Add new pluging for conditional update that makes mutation available
Jan 10, 2025
keith-turner
changed the title
Add new pluging for conditional update that makes mutation available
Add new plugin for conditional update that makes mutation available
Jan 11, 2025
keith-turner
added a commit
to keith-turner/accumulo
that referenced
this issue
Jan 11, 2025
This change fixes apache#5188. Unfortunately it touches a lot of code because of cascading dependencies in the code. It would be difficult to break this into a smaller commit. These changes do reduce some of those dependencies though. There are two major advantages after this change. First tablet metadata is no longer kept in memory for queued compactions. Second the tablet metadata is no longer read during compaction reservation. Before this change the following would happen. 1. TGW would find a tablet to compact and enqueue compaction jobs+tablet metadata. 2. Eventually when a compactor requested a job it would yank job+tablet metadata off the queue. 3. To reserve the compaction a lot of complex analysis was done in the coordinator and then a conditional mutation was submitted. The conditional mutation would require all data involved in the complex analysis to be the same. 4. If the conditional mutation failed then the coordinator would reread the tablet metadata and go back to step 3. After this change the following happens in the code. 1. TGW would find a tablet to compact and enqueue a new class called ResolvedCompactionJob. This new class takes the compaction job and tablet metadata and computes all information needed for the compaction later. The TabletMetadata object is no longer refrenced by this class after the constructor returns. So this class will use much less memory on the queue for the case when tablet have lots of files. 2. Eventually when a compactor requested a job it would yank a ResolvedCompactionJob off the queue. 3. All of the complex analysis to determine if a compaction can start is now done in the conditional mutation instead of in the coordinator. To enable this, new functionality was added to Ample including the new TabletMetadataCheck interface, TabletMetadataCheckIterator, and the CompactionReservationCheck class. With these changes its now super easy to write a conditional check that will do arbitrary analysis of TabletMetadata prior to committing a mutation. 4. Since the analysis is done in the conditional mutation there is no longer a need to retry. If the mutation fails then we know the compaction can not run. The following were some supporting changes that had to be made. * Took methods for encoding KeyExtent as base64 from TabletManagementParameters and moved the KeyExtent because this was needed in the new TabletMetadataCheckIterator. * Move TabletMetadata out of the compaction queues, which made those more independent but was a big change. The main change here is that instead of adding `TabletMetadata, List<CompactionJob>` to the compaction queue now `KeyExtent, List<CompactionJob>` is added. This required changing the test for these classes and the code that interacts with them in CompactionCoordinator creating lots of diff. * Moved CompactionCoordinatorTest.testCanReserve() to CompactionReservationCheckTest.testCanReserve() and changed the code to work with the new CompactionReservationCheck class. These are test of the complex logic that used to run in the coordinator and now runs in the tablet server as part of a conditional mutation. * Removed conditional checks from Ample related to compactions that were no longer used after these changes. There was code for requiring a set of files not not be compacting. The new TabletMetadataCheck functionality of ample could be used to simplify other conditional mutation checks of tablet metadata. It made the compaction reservation code much simpler and easier to understand. This code could be further improved if apache#5244 were implemented.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
Conditional updates allow atomically inspecting a row in a tablet server before making an update. Currently this inspection can be done via checking that columns have exact values. Optionally an iterator can be run as part of the check to perform more complex analysis. When using iterators for more complex analysis the server side iterator code can inspect the current row data, however it does not know what updates are attempting to be made to the row.
When using conditional updates for the accumulo metadata table in 4.0 the following pattern has emerged.
In the above situation encoding the information twice is error prone and a waste of time for the developer and the runtime code. For the case of accumulo metadata updates that add a compaction to a tablet metadata row it currently does something like the following.
This is an example of the duplicate encoding that is being done.
Describe the solution you'd like
A new server side plugin that can be executed as part of a conditional update that has access to at least the following two things.
This new ConditionalCheck could specified to run for a conditional mutation. It would have access to the current row data and the updates that are being attempted for the row. This would avoid the duplicate enoding issue.
The text was updated successfully, but these errors were encountered: