[Bugfix][Mamba] Fix Multistep on Mamba-like models #10705
Conversation
ids are not passed correctly to the mamba cache manager, thus doesn't release mamba cache slots on time Signed-off-by: mzusman <[email protected]>
mzusman
requested review from
DarkLight1337,
ywang96,
zhuohan123,
youkaichao,
alexm-redhat,
comaniac and
njhill
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
Signed-off-by: mzusman <[email protected]>
Signed-off-by: mzusman <[email protected]>
| ) -> None: | ||
| # This test is verifying that multistep works correctly | ||
| #on mamba-like models | ||
| with vllm_runner(model, num_scheduler_steps=8, |
There was a problem hiding this comment.
It would be nice to add check to make sure that the mamba cache is empty at the end of the test to catch #10693, but unfortunately I don't see a way to do it that's not very high effort.
Signed-off-by: mzusman <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>
Signed-off-by: mzusman <[email protected]>
Signed-off-by: mzusman <[email protected]>
Bugfix for Mamba with multistep scheduler, where finished requests ids were not passed into the MambaCacheManager,
which in turn ran into out of free slots situation, unable to receive new incoming requests,
The fix was to move the get_and_reset_finished_request_call inside the if clause so it would be called only upon finished multistep run.
FIX #10693
CC @tlrmchlsmth , @fabianlim