Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fold all conditions into the pending workload query #18

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

julik
Copy link
Contributor

@julik julik commented Sep 26, 2024

When we autoscale based on queue depth, and use scheduler keys and fuses, we want to take into account all the conditions. The idea has two benefits:

  • We should have only 1 scope for selecting the next workload, without having it double. It should be sufficiently efficient to be usable in both contexts (autoscaler metric and dequeue)
  • When we autoscale, the fuses should have effect on that autoscaling. If we fuse out 80% of the workloads, there is no point running 80% of compute allocated to those workloads in the anticipation that they are going to become needed once we de-fuse the workloads back in.

Same for scheduler keys - it is possible to know which scheduler keys are currently known to the system, and there is no reason removed scheduled workloads should still influence the autoscaler.

This is WIP, as we need a decent benchmark which allows us to tweak this scope/query with a realistic queue size matching our real-life setup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant