LoRA Support for Ultravox model #11253

thedebugger · 2024-12-17T06:17:41Z

This should also work w/ mistral models which also uses LlamaForCasualLM architecture re: here

github-actions · 2024-12-17T06:18:13Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

thedebugger · 2024-12-17T06:29:06Z

Hi folks, I'm working with @petersalas on this. PR is not complete but wanted to start the discussion as I have some open questions and need some help from vLLM community

vllm/model_executor/models/ultravox.py

tests/lora/conftest.py

mergify · 2024-12-31T17:48:45Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @thedebugger.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

tests/lora/conftest.py

Signed-off-by: Sumit Vij <[email protected]>

WIP: lora tests Minor tweaks Moar fixes Temp changes Cleanup Add more debugging logs and packed modules Signed-off-by: Sumit Vij <[email protected]>

Remove stale comment Add llama lora modules Add llama test case Add test case and log warning on missing lora modules Rollback unwanted changes and format fixes Signed-off-by: Sumit Vij <[email protected]>

jeejeelee · 2025-01-02T10:13:39Z

Can you refer to #10022 to minimize the changes?

thedebugger · 2025-01-02T13:37:05Z

Can you refer to #10022 to minimize the changes?

Changes are inline with 10022 except the test case and other minor logging changes. Do you have any concerns with any particular change?

jeejeelee · 2025-01-02T15:05:29Z

It looks like there are issues with both the added tests and logs. We should only modify the Ultravox scipt, following the changes made in the #10022

thedebugger · 2025-01-02T16:48:48Z

What is/are the issue(s)? Maybe I miss something but tests are passing

thedebugger · 2025-01-04T13:13:02Z

@jeejeelee lmk what are your concerns please? Happy to address it. Having a test case was super helpful to make sure LoRA works as expected with llama and ultravox

vllm/lora/models.py

tests/lora/conftest.py

…-ultravox-lora-dec-16

Signed-off-by: Jee Jee Li <[email protected]>

jeejeelee · 2025-01-06T10:37:44Z

@thedebugger We should only modify ultravox.py, please revert other changes. After you revert them, we can merge this PR.

thedebugger · 2025-01-13T04:27:02Z

@jeejeelee does the PR now looks okay to you? The changes are only related to ultravox and ultravox test. And, I'll open up another PR related to missing lora modules warning message

Note: AFAICT, CI and GPU tests are failing for unrelated reason which I also see happening it in other PRs that have been merged into main

Signed-off-by: Sumit Vij <[email protected]>

jeejeelee · 2025-01-13T08:29:50Z

tests/conftest.py

@@ -858,7 +865,8 @@ def generate_greedy_logprobs(
                                        greedy_logprobs_params,
                                        images=images,
                                        audios=audios,
-                                        videos=videos)
+                                        videos=videos,
+                                        **kwargs)

    def generate_encoder_decoder_greedy_logprobs(
        self,


The changes to this file are not related to this PR, please revert.

tests/lora/test_ultravox.py

vllm/model_executor/models/ultravox.py

jeejeelee · 2025-01-13T08:33:49Z

@jeejeelee does the PR now looks okay to you? The changes are only related to ultravox and ultravox test. And, I'll open up another PR related to missing lora modules warning message

Note: AFAICT, CI and GPU tests are failing for unrelated reason which I also see happening it in other PRs that have been merged into main

After completing the above modifications, this PR can be merged, thanks

…-ultravox-lora-dec-16

Signed-off-by: Jee Jee Li <[email protected]>

…-ultravox-lora-dec-16

Signed-off-by: Jee Jee Li <[email protected]>

thedebugger · 2025-01-16T22:40:04Z

@petersalas does PR look good to you? If yes, please approve

petersalas

LGTM

thedebugger · 2025-01-17T01:40:21Z

@jeejeelee okay to approve and merge?

jeejeelee

Let's wait for the lora test issue mentioned in #12111 to be resolved before considering starting the merge testing.

thedebugger · 2025-01-17T05:01:42Z

~~Do you know when does lora test run to make sure it is working fine in CI? I can't find it running in some of buildkite CI jobs~~ Nm, I found the info

…-ultravox-lora-dec-16

Signed-off-by: Jee Jee Li <[email protected]>

jeejeelee

Let's keep the unit tests as they are for now. I'll look into how to train Ultravox LoRA and update in another PR

thedebugger · 2025-01-20T07:25:10Z

@jeejeelee I'll fix the test tomorrow. I pulled up the latest code and it is failing for me with Cuda OOM (different than CI failure). I also ran the test again on 208e662, it works. So likely something changed on master that is causing test to fail.

AFAICT, the failure is not related to LoRA, so training ultravox lora shouldn't have any impact. Let me troubleshoot this tomorrow and figure out what is going on before you spend time on this.

thedebugger · 2025-01-21T07:21:30Z

I spent more time looking at the failure. The failure happens during init when running ultravox with dummy data and I haven't been able to reproduce it locally (though locally test fails with cuda oom error for llama on latest version). I'll look further tomorrow why CI is seeing device mismatch when running whisper. I checked the other ultravox test, that is working fine. I'll look further into it tomorrow

jeejeelee · 2025-01-21T07:33:54Z

I spent more time looking at the failure. The failure happens during init when running ultravox with dummy data and I haven't been able to reproduce it locally (though locally test fails with cuda oom error for llama on latest version). I'll look further tomorrow why CI is seeing device mismatch when running whisper. I checked the other ultravox test, that is working fine. I'll look further into it tomorrow

We can first remove the LoRA test-related code and merge this PR. I'll spend some time later training a LoRA model. What do you think?

Signed-off-by: Sumit Vij <[email protected]> Reduce model len Signed-off-by: Sumit Vij <[email protected]>

DarkLight1337 requested a review from jeejeelee December 17, 2024 06:19

jeejeelee reviewed Dec 17, 2024

View reviewed changes

vllm/model_executor/models/ultravox.py Show resolved Hide resolved

thedebugger commented Dec 17, 2024

View reviewed changes

vllm/model_executor/models/ultravox.py Show resolved Hide resolved

jeejeelee reviewed Dec 17, 2024

View reviewed changes

vllm/model_executor/models/ultravox.py Outdated Show resolved Hide resolved

vllm/model_executor/models/ultravox.py Outdated Show resolved Hide resolved

thedebugger commented Dec 17, 2024

View reviewed changes

tests/lora/conftest.py Outdated Show resolved Hide resolved

mergify bot added the needs-rebase label Dec 31, 2024

thedebugger commented Dec 31, 2024

View reviewed changes

tests/lora/conftest.py Outdated Show resolved Hide resolved

thedebugger force-pushed the svij-ultravox-lora-dec-16 branch 2 times, most recently from 771484d to 64a664f Compare December 31, 2024 18:49

mergify bot removed the needs-rebase label Dec 31, 2024

thedebugger changed the title ~~WIP: Ultravox Support for LoRA~~ Ultravox Support for LoRA Dec 31, 2024

thedebugger marked this pull request as ready for review December 31, 2024 18:52

thedebugger added 3 commits January 1, 2025 09:30

WIP: early draft of lora support in Ultravox

1c55938

Signed-off-by: Sumit Vij <[email protected]>

format fixes

5a6b79f

WIP: lora tests Minor tweaks Moar fixes Temp changes Cleanup Add more debugging logs and packed modules Signed-off-by: Sumit Vij <[email protected]>

Fix lora modules and formatting

3f5996c

Remove stale comment Add llama lora modules Add llama test case Add test case and log warning on missing lora modules Rollback unwanted changes and format fixes Signed-off-by: Sumit Vij <[email protected]>

thedebugger force-pushed the svij-ultravox-lora-dec-16 branch from 64a664f to 3f5996c Compare January 1, 2025 17:31

jeejeelee reviewed Jan 6, 2025

View reviewed changes

vllm/lora/models.py Outdated Show resolved Hide resolved

jeejeelee reviewed Jan 6, 2025

View reviewed changes

tests/lora/conftest.py Outdated Show resolved Hide resolved

jeejeelee added 3 commits January 6, 2025 03:21

Merge branch 'main' of https://github.com/vllm-project/vllm into svij…

d1b65eb

…-ultravox-lora-dec-16

Merge branch 'main' of https://github.com/vllm-project/vllm into svij…

7367bc2

…-ultravox-lora-dec-16

Done

2abf2ab

Signed-off-by: Jee Jee Li <[email protected]>

Fix formatting and test case

4a633d3

Signed-off-by: Sumit Vij <[email protected]>

thedebugger force-pushed the svij-ultravox-lora-dec-16 branch from dd686e3 to 4a633d3 Compare January 13, 2025 04:38

jeejeelee reviewed Jan 13, 2025

View reviewed changes

tests/lora/test_ultravox.py Show resolved Hide resolved

jeejeelee reviewed Jan 13, 2025

View reviewed changes

vllm/model_executor/models/ultravox.py Show resolved Hide resolved

jeejeelee added 4 commits January 16, 2025 02:21

Merge branch 'main' of https://github.com/vllm-project/vllm into svij…

224a65e

…-ultravox-lora-dec-16

Done

769f7bd

Signed-off-by: Jee Jee Li <[email protected]>

Merge branch 'main' of https://github.com/vllm-project/vllm into svij…

907b3c7

…-ultravox-lora-dec-16

Add doc

208e662

Signed-off-by: Jee Jee Li <[email protected]>

mergify bot added the documentation Improvements or additions to documentation label Jan 16, 2025

petersalas approved these changes Jan 17, 2025

View reviewed changes

jeejeelee approved these changes Jan 17, 2025

View reviewed changes

Merge branch 'main' of https://github.com/vllm-project/vllm into svij…

1248d5f

…-ultravox-lora-dec-16

jeejeelee added the ready ONLY add when PR is ready to merge/full CI is needed label Jan 18, 2025

jeejeelee enabled auto-merge (squash) January 18, 2025 16:01

jeejeelee added 3 commits January 19, 2025 01:57

Merge branch 'main' of https://github.com/vllm-project/vllm into svij…

575b5dc

…-ultravox-lora-dec-16

Merge branch 'main' of https://github.com/vllm-project/vllm into svij…

7cb7eba

…-ultravox-lora-dec-16

Optmize unit test

f483d9a

Signed-off-by: Jee Jee Li <[email protected]>

jeejeelee approved these changes Jan 20, 2025

View reviewed changes

auto-merge was automatically disabled January 22, 2025 06:29
Head branch was pushed to by a user without write access

Test setting cpu as a default device

1195ad8

Signed-off-by: Sumit Vij <[email protected]> Reduce model len Signed-off-by: Sumit Vij <[email protected]>

thedebugger force-pushed the svij-ultravox-lora-dec-16 branch from d247036 to 1195ad8 Compare January 22, 2025 07:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LoRA Support for Ultravox model #11253

LoRA Support for Ultravox model #11253

thedebugger commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024

thedebugger commented Dec 17, 2024 •

edited

Loading

mergify bot commented Dec 31, 2024

jeejeelee commented Jan 2, 2025

thedebugger commented Jan 2, 2025

jeejeelee commented Jan 2, 2025

thedebugger commented Jan 2, 2025

thedebugger commented Jan 4, 2025

jeejeelee commented Jan 6, 2025

thedebugger commented Jan 13, 2025 •

edited

Loading

jeejeelee Jan 13, 2025

jeejeelee commented Jan 13, 2025

thedebugger commented Jan 16, 2025

petersalas left a comment

thedebugger commented Jan 17, 2025

jeejeelee left a comment

thedebugger commented Jan 17, 2025 •

edited

Loading

jeejeelee left a comment •

edited

Loading

thedebugger commented Jan 20, 2025 •

edited

Loading

thedebugger commented Jan 21, 2025 •

edited

Loading

jeejeelee commented Jan 21, 2025

LoRA Support for Ultravox model #11253

Are you sure you want to change the base?

LoRA Support for Ultravox model #11253

Conversation

thedebugger commented Dec 17, 2024 • edited Loading

github-actions bot commented Dec 17, 2024

thedebugger commented Dec 17, 2024 • edited Loading

mergify bot commented Dec 31, 2024

jeejeelee commented Jan 2, 2025

thedebugger commented Jan 2, 2025

jeejeelee commented Jan 2, 2025

thedebugger commented Jan 2, 2025

thedebugger commented Jan 4, 2025

jeejeelee commented Jan 6, 2025

thedebugger commented Jan 13, 2025 • edited Loading

jeejeelee Jan 13, 2025

Choose a reason for hiding this comment

jeejeelee commented Jan 13, 2025

thedebugger commented Jan 16, 2025

petersalas left a comment

Choose a reason for hiding this comment

thedebugger commented Jan 17, 2025

jeejeelee left a comment

Choose a reason for hiding this comment

thedebugger commented Jan 17, 2025 • edited Loading

jeejeelee left a comment • edited Loading

Choose a reason for hiding this comment

thedebugger commented Jan 20, 2025 • edited Loading

thedebugger commented Jan 21, 2025 • edited Loading

jeejeelee commented Jan 21, 2025

thedebugger commented Dec 17, 2024 •

edited

Loading

thedebugger commented Dec 17, 2024 •

edited

Loading

thedebugger commented Jan 13, 2025 •

edited

Loading

thedebugger commented Jan 17, 2025 •

edited

Loading

jeejeelee left a comment •

edited

Loading

thedebugger commented Jan 20, 2025 •

edited

Loading

thedebugger commented Jan 21, 2025 •

edited

Loading