Implement priority queueing in the resource syncer #1052
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tpantelis
requested review from
Oats87,
skitt,
sridhargaddam and
vthapar
as code owners
|
🤖 Created branch: z_pr1052/tpantelis/priority_queue |
tpantelis
force-pushed
the
priority_queue
branch
3 times, most recently
from
6d0f9cf to
1497d90
Compare
...for use with the "container/heap" module that implements heap.Interface. The items in the queue are ordered by descending priority such that items with higher priority values are de-queued first. Signed-off-by: Tom Pantelis <[email protected]>
Signed-off-by: Tom Pantelis <[email protected]>
Signed-off-by: Tom Pantelis <[email protected]>
...by configuring backend Queue for the client-go rate-limiting queue. The work queue exposes two priorities for enqueue: normal priority via the existing Enqueue method and low priority via a new EnqueueLowPriority method. Signed-off-by: Tom Pantelis <[email protected]>
Signed-off-by: Tom Pantelis <[email protected]>
Utilize the workqueue's EnqueueLowPriority method to prioritize newly created/updated resources over pre-existing resources on startup. The cache.ResourceEventHandler's OnAdd method has an 'isInInitialList' param that indicates if the obj is pre-existing in the initial listing retrieved from the API server. If 'isInInitialList' is true then enqueue with low priority otherwise use normal priority (Enqueue method). The underlying priority queue might adjust the ordering of items with the same priority so the ordering of items enqueued might not be the same order that they're dequeued. As such, the namespace event handling needed to be modified to use the 'operationQueues' to maintain the ordering of adds and deletes. Signed-off-by: Tom Pantelis <[email protected]>
On an informer re-sync, all resources are re-notified as update events. Most likely the resources didn't actually change so enqueue with low priority if the ResourceVersion didn't change. Signed-off-by: Tom Pantelis <[email protected]>
tpantelis
force-pushed
the
priority_queue
branch
from
1497d90 to
3a79f6a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
All pre-existing resources are queued by the informer on startup. With many resources in the hundreds or thousands it can take significant time to process them all and the work queue rate limiter can cause further significant delays. This can block newly created resources or existing resource updates for potentially minutes. Most of the time a pod restart is fairly quick so most resources didn't likely change so it makes sense to give newly created/updated resources higher priority and process them in a more timely manner.
The client-go work queue allows the backend
Queueimplementation to be configurable. This PR adds an implementation to the admiralworkqueuemodule that is backed by a priority queue that is used in conjunction with thecontainer/heapmodule and implementsheap.Interface. A newEnqueueLowPrioritymethod was added to the admiralworkqueue.Interfacethat enqueues the item with priority 0. The existingEnqueuemethod was modified to enqueue with higher priority 1.The resource syncer was modified to use the
EnqueueLowPrioritymethod to prioritize newly created/updated resources over pre-existing resources on startup. Thecache.ResourceEventHandler'sOnAddmethod has anisInInitialListparam that indicates if the obj was pre-existing in the initial listing retrieved from the API server. IfisInInitialListis true then enqueue with low priority otherwise use normal priority (Enqueuemethod).See commits for more details.
Related to submariner-io/lighthouse#1706