-
-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[V1] Do not allocate beyond the max_model_len #10730
Conversation
Signed-off-by: Woosuk Kwon <[email protected]>
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Otherwise LGTM. Also where do we reject too long requests? Seems like the scheduler doesn't do that?
@comaniac You're right. V1 doesn't have it now. |
I see. We could add that later. Meanwhile could you add a TODO in this PR as a reminder? |
Signed-off-by: Woosuk Kwon <[email protected]>
@comaniac Good point. Added. PTAL. |
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
This PR fixes a bug in V1 KV cache manager that allocates more than
max_model_len // block_size
blocks for a request. This is not only inefficient but also illegal because the block table has a fixed shape of[max_num_reqs, max_model_len // block_size]
.