Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

Commit

Permalink
format
Browse files Browse the repository at this point in the history
  • Loading branch information
@robertgshaw2-redhat
robertgshaw2-redhat committed Jun 26, 2024
1 parent ff0ea23 commit 95eb999
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion tests/accuracy/test_lm_eval_correctness.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@
reason="lm_eval required")

RTOL = 0.02
TEST_DATA_FILE = os.environ.get("LM_EVAL_TEST_DATA_FILE", None)
TEST_DATA_FILE = os.environ.get(
"LM_EVAL_TEST_DATA_FILE",
".github/lm-eval-configs/models/Meta-Llama-3-8B-Instruct.yaml")


def wait_for_server(timeout=900) -> bool:
Expand Down

1 comment on commit 95eb999

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bigger_is_better

Benchmark suite Current: 95eb999 Previous: 93b1bb9 Ratio
{"name": "request_throughput", "description": "VLLM Engine throughput - synthetic\nmodel - NousResearch/Llama-2-7b-chat-hf\nmax_model_len - 4096\nbenchmark_throughput {\n \"use-all-available-gpus_\": \"\",\n \"input-len\": 256,\n \"output-len\": 128,\n \"num-prompts\": 1000\n}", "gpu_description": "NVIDIA L4 x 1", "vllm_version": "0.5.1", "python_version": "3.10.12 (main, Jun 7 2023, 13:43:11) [GCC 11.3.0]", "torch_version": "2.3.0+cu121"} 2.4779059565479895 prompts/s 2.4872798902675823 prompts/s 1.00
{"name": "token_throughput", "description": "VLLM Engine throughput - synthetic\nmodel - NousResearch/Llama-2-7b-chat-hf\nmax_model_len - 4096\nbenchmark_throughput {\n \"use-all-available-gpus_\": \"\",\n \"input-len\": 256,\n \"output-len\": 128,\n \"num-prompts\": 1000\n}", "gpu_description": "NVIDIA L4 x 1", "vllm_version": "0.5.1", "python_version": "3.10.12 (main, Jun 7 2023, 13:43:11) [GCC 11.3.0]", "torch_version": "2.3.0+cu121"} 951.515887314428 tokens/s 955.1154778627516 tokens/s 1.00

This comment was automatically generated by workflow using github-action-benchmark.

Please sign in to comment.