After using llmcompressor to create a partially 2:4 sparse FP8 quantized model (where the MLP layers are sparse, but the attention layers are not), I tested its speed and found that the inference speed of this model did not differ from that of a regular FP8 quantized model. Further investigation revealed that although the MLP layers are 2:4 sparse, the get_scheme function in the CompressedTensorsConfig of the code repository does not handle this partially sparse model appropriately, causing the MLP layers to not utilize the CompressedTensors24 scheme.
Before:

Fixed:

FIX vllm-project/llm-compressor#1037

Tested_model:

• original qwen2.5-3B
• fp8 quantized qwen2.5-3B
• fully 2:4 sparse fp8 quantized qwen2.5-3B
• partially 2:4 sparse fp8 quantized qwen2.5-3B