[SGLang] Wave backend fails accuracy threshold for Llama3.1 model on `mmlu` eval
- Dominant language
- Python
- Stars
- 59
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
As stated in the title, Wave backend fails to meet the accuracy threshold when running Llama3.1B model on `mmlu` eval dataset. Note that this is with cuda graphs turned off. This can be reproduced by adding the following test function in `test_wave_attention_backend.py` (see test_triton_attention_backend.py` for test parity we'd like to hit)
Test function:
```
def test_mmlu(self):
model = DEFAULT_MODEL_NAME_FOR_TEST
base_url = DEFAULT_URL_FOR_TEST
process = popen_launch_server(
model,
base_url,
timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
other_args=["--attention-backend", "wave", "--enable-torch-compile", "--disable-cuda-graph",],
)
try:
args = SimpleNamespace(
base_url=base_url,
model=model,
eval_name="mmlu",
num_examples=64,
num_threads=32,
)
metrics = run_eval(args)
self.assertGreaterEqual(metrics["score"], 0.65)
finally:
kill_process_tree(process.pid)
```
Error:
```
metrics = run_eval(args)
> self.assertGreaterEqual(metrics["score"], 0.65)
E AssertionError: 0.0625 not greater than or equal to 0.65
test/srt/test_wave_attention_backend.py:59: AssertionError
```
Contributor guide
Assessment
This issue has not been assessed yet.