[Hardware][CPU] Add embedding models support for CPU backend #10193

Isotr0py · 2024-11-10T09:47:05Z

FIX #9379

Implement CPU embedding model runner
Add encoder-only attention support to SDPA backend

Signed-off-by: Isotr0py <[email protected]>

github-actions · 2024-11-10T09:47:16Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

.buildkite/run-cpu-test.sh

vllm/model_executor/models/bert.py

Signed-off-by: Isotr0py <[email protected]>

tests/models/embedding/language/test_embedding.py

DarkLight1337

Looks good overall, see if the tests pass.

In the future we should also add core_model and cpu_model tags to embedding model tests.

DarkLight1337 · 2024-11-11T09:18:29Z

Oh, the test failed but the CI still passed 😅

Isotr0py · 2024-11-11T09:27:51Z

Ah, the encoder-decoder tests are broken due to runner refactor. Let me make a quick PR to fix it. 😅

Isotr0py · 2024-11-11T09:42:02Z

I have opened #10218 to fix the broken encoder-decoder CPU runner.

…oject#10193) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Jee Jee Li <[email protected]>

…oject#10193) Signed-off-by: Isotr0py <[email protected]>

…oject#10193) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Sumit Dubey <[email protected]>

…oject#10193) Signed-off-by: Isotr0py <[email protected]>

…oject#10193) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Maxime Fournioux <[email protected]>

…oject#10193) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]>

githebs · 2024-11-25T14:47:39Z

Hi everyone @Isotr0py @DarkLight1337
Is it possible to have an example on how to launch vllm on cpu with an embedding model ?
Having a hard time to see how this works

Thanks in advance

DarkLight1337 · 2024-11-25T14:48:40Z

Hi everyone @Isotr0py @DarkLight1337 Is it possible to have an example on how to launch vllm on cpu with an embedding model ? Having a hard time to see how this works

Thanks in advance

You can use vllm serve <model> --task embedding.

githebs · 2024-11-25T14:54:41Z

Thanks for the reply and sorry to bother you for that once more but here is my compose

services:
  vllm:
      container_name: vllm
      image: vllm/vllm-openai:latest
      restart: unless-stopped
      shm_size: '64gb'
      command: --host 0.0.0.0 --model /models/HIT-TMG-KaLM-embedding-multilingual-mini-instruct-v1 --max-model-len 32768 --device cpu --task embedding
      environment:
        - ENGINE_ITERATION_TIMEOUT_S=180
      volumes:
        - /models/:/models
      ports:
        - 9000:8000

got console error

INFO 11-25 06:52:13 api_server.py:585] vLLM API server version 0.6.4.post1
INFO 11-25 06:52:13 api_server.py:586] args: Namespace(host='0.0.0.0', port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/models/HIT-TMG-KaLM-embedding-multilingual-mini-instruct-v1', task='embedding', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', chat_template_text_format='string', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=32768, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='cpu', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 11-25 06:52:13 api_server.py:175] Multiprocessing frontend to use ipc:///tmp/b0f1e755-446e-4ad2-9e5e-23bd11657abc for IPC Path.
INFO 11-25 06:52:13 api_server.py:194] Started engine process with PID 16
INFO 11-25 06:52:13 config.py:1861] Downcasting torch.float32 to torch.float16.
INFO 11-25 06:52:17 config.py:1861] Downcasting torch.float32 to torch.float16.
WARNING 11-25 06:52:17 arg_utils.py:1075] [DEPRECATED] Block manager v1 has been removed, and setting --use-v2-block-manager to True or False has no effect on vLLM behavior. Please remove --use-v2-block-manager in your engine argument. If your use case is not supported by SelfAttnBlockSpaceManager (i.e. block manager v2), please file an issue with detailed information.
WARNING 11-25 06:52:17 config.py:487] Async output processing is only supported for CUDA, TPU, XPU and HPU.Disabling it for other platforms.
WARNING 11-25 06:52:20 arg_utils.py:1075] [DEPRECATED] Block manager v1 has been removed, and setting --use-v2-block-manager to True or False has no effect on vLLM behavior. Please remove --use-v2-block-manager in your engine argument. If your use case is not supported by SelfAttnBlockSpaceManager (i.e. block manager v2), please file an issue with detailed information.
WARNING 11-25 06:52:20 config.py:487] Async output processing is only supported for CUDA, TPU, XPU and HPU.Disabling it for other platforms.
INFO 11-25 06:52:20 llm_engine.py:249] Initializing an LLM engine (v0.6.4.post1) with config: model='/models/HIT-TMG-KaLM-embedding-multilingual-mini-instruct-v1', speculative_config=None, tokenizer='/models/HIT-TMG-KaLM-embedding-multilingual-mini-instruct-v1', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/models/HIT-TMG-KaLM-embedding-multilingual-mini-instruct-v1, num_scheduler_steps=1, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=False, use_cached_outputs=True, chat_template_text_format=string, mm_processor_kwargs=None, pooler_config=PoolerConfig(pooling_type='MEAN', normalize=True, softmax=None, step_tag_id=None, returned_token_ids=None))
WARNING 11-25 06:52:20 cpu_executor.py:320] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 11-25 06:52:20 cpu_executor.py:350] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 11-25 06:52:20 selector.py:261] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 11-25 06:52:20 selector.py:144] Using XFormers backend.
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
ERROR 11-25 06:52:24 engine.py:366] There is no module or parameter named 'embed_tokens' in Qwen2EmbeddingModel
ERROR 11-25 06:52:24 engine.py:366] Traceback (most recent call last):
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
ERROR 11-25 06:52:24 engine.py:366]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 11-25 06:52:24 engine.py:366]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
ERROR 11-25 06:52:24 engine.py:366]     return cls(ipc_path=ipc_path,
ERROR 11-25 06:52:24 engine.py:366]            ^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 71, in __init__
ERROR 11-25 06:52:24 engine.py:366]     self.engine = LLMEngine(*args, **kwargs)
ERROR 11-25 06:52:24 engine.py:366]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 347, in __init__
ERROR 11-25 06:52:24 engine.py:366]     self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 11-25 06:52:24 engine.py:366]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 36, in __init__
ERROR 11-25 06:52:24 engine.py:366]     self._init_executor()
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/cpu_executor.py", line 121, in _init_executor
ERROR 11-25 06:52:24 engine.py:366]     self._run_workers("load_model")
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/cpu_executor.py", line 181, in _run_workers
ERROR 11-25 06:52:24 engine.py:366]     driver_worker_output = self.driver_method_invoker(
ERROR 11-25 06:52:24 engine.py:366]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/cpu_executor.py", line 373, in _driver_method_invoker
ERROR 11-25 06:52:24 engine.py:366]     return getattr(driver, method)(*args, **kwargs)
ERROR 11-25 06:52:24 engine.py:366]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/cpu_worker.py", line 206, in load_model
ERROR 11-25 06:52:24 engine.py:366]     self.model_runner.load_model()
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/cpu_model_runner.py", line 469, in load_model
ERROR 11-25 06:52:24 engine.py:366]     self.model = get_model(vllm_config=self.vllm_config)
ERROR 11-25 06:52:24 engine.py:366]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
ERROR 11-25 06:52:24 engine.py:366]     return loader.load_model(vllm_config=vllm_config)
ERROR 11-25 06:52:24 engine.py:366]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 334, in load_model
ERROR 11-25 06:52:24 engine.py:366]     model.load_weights(self._get_all_weights(model_config, model))
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 570, in load_weights
ERROR 11-25 06:52:24 engine.py:366]     loader.load_weights(weights)
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 229, in load_weights
ERROR 11-25 06:52:24 engine.py:366]     autoloaded_weights = list(self._load_module("", self.module, weights))
ERROR 11-25 06:52:24 engine.py:366]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 11-25 06:52:24 engine.py:366]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 218, in _load_module
ERROR 11-25 06:52:24 engine.py:366]     raise ValueError(msg)
**_ERROR 11-25 06:52:24 engine.py:366] ValueError: There is no module or parameter named 'embed_tokens' in Qwen2EmbeddingModel_**
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 368, in run_mp_engine
    raise e
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
    return cls(ipc_path=ipc_path,
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 71, in __init__
    self.engine = LLMEngine(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 347, in __init__
    self.model_executor = executor_class(vllm_config=vllm_config, )
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 36, in __init__
    self._init_executor()
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/cpu_executor.py", line 121, in _init_executor
    self._run_workers("load_model")
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/cpu_executor.py", line 181, in _run_workers
    driver_worker_output = self.driver_method_invoker(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/cpu_executor.py", line 373, in _driver_method_invoker
    return getattr(driver, method)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/worker/cpu_worker.py", line 206, in load_model
    self.model_runner.load_model()
  File "/usr/local/lib/python3.12/dist-packages/vllm/worker/cpu_model_runner.py", line 469, in load_model
    self.model = get_model(vllm_config=self.vllm_config)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
    return loader.load_model(vllm_config=vllm_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 334, in load_model
    model.load_weights(self._get_all_weights(model_config, model))
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 570, in load_weights
    loader.load_weights(weights)
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 229, in load_weights
    autoloaded_weights = list(self._load_module("", self.module, weights))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 218, in _load_module
    raise ValueError(msg)
ValueError: There is no module or parameter named 'embed_tokens' in Qwen2EmbeddingModel
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 643, in <module>
    uvloop.run(run_server(args))
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 609, in run_server
    async with build_async_engine_client(args) as engine_client:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 113, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 210, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.

Any hint ?

DarkLight1337 · 2024-11-25T16:05:19Z

Please try #10639 and see if it fixes your issue.

…oject#10193) Signed-off-by: Isotr0py <[email protected]>

Isotr0py added 6 commits November 10, 2024 14:29

init cpu embedding runner

b4e50cb

Signed-off-by: Isotr0py <[email protected]>

sdpa backend support encoder-only

2cf78c5

Signed-off-by: Isotr0py <[email protected]>

code format

a2bb6c4

Signed-off-by: Isotr0py <[email protected]>

refactor CPU model runner

232fd43

Signed-off-by: Isotr0py <[email protected]>

add necessary comments

dd89c30

Signed-off-by: Isotr0py <[email protected]>

enable cpu embedding tests

f0ff16a

Signed-off-by: Isotr0py <[email protected]>

mergify bot added the ci/build label Nov 10, 2024

Isotr0py added the x86 CPU label Nov 10, 2024

DarkLight1337 self-assigned this Nov 10, 2024

DarkLight1337 reviewed Nov 10, 2024

View reviewed changes

.buildkite/run-cpu-test.sh Show resolved Hide resolved

vllm/model_executor/models/bert.py Show resolved Hide resolved

fix cpu tests

184055d

Signed-off-by: Isotr0py <[email protected]>

Isotr0py marked this pull request as ready for review November 11, 2024 06:09

Isotr0py requested a review from ywang96 as a code owner November 11, 2024 06:09

Isotr0py commented Nov 11, 2024

View reviewed changes

tests/models/embedding/language/test_embedding.py Show resolved Hide resolved

DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 11, 2024

DarkLight1337 approved these changes Nov 11, 2024

View reviewed changes

DarkLight1337 enabled auto-merge (squash) November 11, 2024 07:24

DarkLight1337 merged commit 58170d6 into vllm-project:main Nov 11, 2024
66 of 67 checks passed

Isotr0py deleted the cpu-embed branch November 11, 2024 09:13

Isotr0py mentioned this pull request Nov 11, 2024

[Bugfix][Hardware][CPU] Fix broken encoder-decoder CPU runner #10218

Merged

jeejeelee pushed a commit to jeejeelee/vllm that referenced this pull request Nov 11, 2024

[Hardware][CPU] Add embedding models support for CPU backend (vllm-pr…

b65a229

…oject#10193) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Jee Jee Li <[email protected]>

rickyyx pushed a commit to rickyyx/vllm that referenced this pull request Nov 13, 2024

[Hardware][CPU] Add embedding models support for CPU backend (vllm-pr…

6bcf00b

…oject#10193) Signed-off-by: Isotr0py <[email protected]>

sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024

[Hardware][CPU] Add embedding models support for CPU backend (vllm-pr…

2441c74

…oject#10193) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Sumit Dubey <[email protected]>

KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024

[Hardware][CPU] Add embedding models support for CPU backend (vllm-pr…

638a538

…oject#10193) Signed-off-by: Isotr0py <[email protected]>

mfournioux pushed a commit to mfournioux/vllm that referenced this pull request Nov 20, 2024

[Hardware][CPU] Add embedding models support for CPU backend (vllm-pr…

31e3447

…oject#10193) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Maxime Fournioux <[email protected]>

tlrmchlsmth pushed a commit to neuralmagic/vllm that referenced this pull request Nov 23, 2024

[Hardware][CPU] Add embedding models support for CPU backend (vllm-pr…

36682f7

…oject#10193) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]>

DarkLight1337 mentioned this pull request Nov 25, 2024

[Model] Enable optional prefix when loading embedding models #10639

Merged

sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024

[Hardware][CPU] Add embedding models support for CPU backend (vllm-pr…

60e575d

…oject#10193) Signed-off-by: Isotr0py <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Hardware][CPU] Add embedding models support for CPU backend #10193

[Hardware][CPU] Add embedding models support for CPU backend #10193

Isotr0py commented Nov 10, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Nov 10, 2024

DarkLight1337 left a comment

DarkLight1337 commented Nov 11, 2024

Isotr0py commented Nov 11, 2024

Isotr0py commented Nov 11, 2024

githebs commented Nov 25, 2024

DarkLight1337 commented Nov 25, 2024

githebs commented Nov 25, 2024

DarkLight1337 commented Nov 25, 2024 •

edited

Loading

[Hardware][CPU] Add embedding models support for CPU backend #10193

[Hardware][CPU] Add embedding models support for CPU backend #10193

Conversation

Isotr0py commented Nov 10, 2024 • edited by github-actions bot Loading

github-actions bot commented Nov 10, 2024

DarkLight1337 left a comment

Choose a reason for hiding this comment

DarkLight1337 commented Nov 11, 2024

Isotr0py commented Nov 11, 2024

Isotr0py commented Nov 11, 2024

githebs commented Nov 25, 2024

DarkLight1337 commented Nov 25, 2024

githebs commented Nov 25, 2024

DarkLight1337 commented Nov 25, 2024 • edited Loading

Isotr0py commented Nov 10, 2024 •

edited by github-actions bot

Loading

DarkLight1337 commented Nov 25, 2024 •

edited

Loading