[1/N] Initial prototype for multi-modal processor

[DarkLight1337]

This PR adds the core code for multi-modal processor while maintaining backward compatibility. The main purpose of this PR is to "reserve" code changes (mainly related to the dependencies of multi-modal processor) to reduce the risk of merge conflicts caused by subsequent PRs.

Note that currently there are no models that use the new multi-modal processor - I will implement a few of them in the next PR. As such, the details of multi-modal processor are still subject to change.

Part of #10114

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. @DarkLight1337 please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

A generalization of our existing code for Phi-3V

[DarkLight1337]

A generalization of our existing code for HF Pixtral

[DarkLight1337]

This was added by the Llama-3.2 PR, but I found that this model doesn't use HF processor in vLLM input processor anymore, so it should be safe to remove.

[Isotr0py]

LGTM!

[Isotr0py]

Seems that v1 entrypoint test is failing: https://buildkite.com/vllm/fastcheck/builds/7789#01931e26-10d6-4960-b792-e949972b2aef/188-1114

[mergify]

rebase

❌ Base branch update has failed

Git reported the following error:

Rebasing (1/7)
Auto-merging tests/models/decoder_only/vision_language/mm_processor_kwargs/test_qwen.py
CONFLICT (content): Merge conflict in tests/models/decoder_only/vision_language/mm_processor_kwargs/test_qwen.py
Auto-merging vllm/config.py
Auto-merging vllm/engine/async_llm_engine.py
Auto-merging vllm/engine/llm_engine.py
Auto-merging vllm/entrypoints/openai/serving_chat.py
Auto-merging vllm/entrypoints/openai/serving_completion.py
Auto-merging vllm/inputs/preprocess.py
Auto-merging vllm/model_executor/models/chatglm.py
Auto-merging vllm/model_executor/models/fuyu.py
Auto-merging vllm/model_executor/models/internvl.py
Auto-merging vllm/model_executor/models/minicpmv.py
Auto-merging vllm/model_executor/models/mllama.py
Auto-merging vllm/model_executor/models/molmo.py
Auto-merging vllm/model_executor/models/pixtral.py
Auto-merging vllm/model_executor/models/qwen.py
Auto-merging vllm/model_executor/models/qwen2_audio.py
Auto-merging vllm/model_executor/models/qwen2_vl.py
Auto-merging vllm/model_executor/models/ultravox.py
Auto-merging vllm/multimodal/__init__.py
CONFLICT (content): Merge conflict in vllm/multimodal/__init__.py
Auto-merging vllm/multimodal/base.py
CONFLICT (content): Merge conflict in vllm/multimodal/base.py
Auto-merging vllm/multimodal/registry.py
CONFLICT (content): Merge conflict in vllm/multimodal/registry.py
Auto-merging vllm/multimodal/video.py
Auto-merging vllm/sequence.py
Auto-merging vllm/spec_decode/draft_model_runner.py
Auto-merging vllm/v1/engine/llm_engine.py
CONFLICT (content): Merge conflict in vllm/v1/engine/llm_engine.py
Auto-merging vllm/worker/cpu_enc_dec_model_runner.py
Auto-merging vllm/worker/cpu_model_runner.py
CONFLICT (content): Merge conflict in vllm/worker/cpu_model_runner.py
Auto-merging vllm/worker/enc_dec_model_runner.py
CONFLICT (content): Merge conflict in vllm/worker/enc_dec_model_runner.py
Auto-merging vllm/worker/model_runner.py
CONFLICT (content): Merge conflict in vllm/worker/model_runner.py
Auto-merging vllm/worker/neuron_model_runner.py
CONFLICT (content): Merge conflict in vllm/worker/neuron_model_runner.py
Auto-merging vllm/worker/openvino_model_runner.py
CONFLICT (content): Merge conflict in vllm/worker/openvino_model_runner.py
Auto-merging vllm/worker/xpu_model_runner.py
CONFLICT (content): Merge conflict in vllm/worker/xpu_model_runner.py
error: could not apply 5108119b... Initial prototype for multi-modal processor
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 5108119b... Initial prototype for multi-modal processor

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @DarkLight1337.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork