iree-org / iree-org/wave

[SGLang] Wave backend fails accuracy threshold for Llama3.1 model on `mmlu` eval

Open
#477 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
59
Forks
32
PR merge metrics
No merged PRs in 30d

Description

As stated in the title, Wave backend fails to meet the accuracy threshold when running Llama3.1B model on `mmlu` eval dataset. Note that this is with cuda graphs turned off. This can be reproduced by adding the following test function in `test_wave_attention_backend.py` (see test_triton_attention_backend.py` for test parity we'd like to hit)

Test function:
```
def test_mmlu(self):
model = DEFAULT_MODEL_NAME_FOR_TEST
base_url = DEFAULT_URL_FOR_TEST
process = popen_launch_server(
model,
base_url,
timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
other_args=["--attention-backend", "wave", "--enable-torch-compile", "--disable-cuda-graph",],
)

try:
args = SimpleNamespace(
base_url=base_url,
model=model,
eval_name="mmlu",
num_examples=64,
num_threads=32,
)

metrics = run_eval(args)
self.assertGreaterEqual(metrics["score"], 0.65)
finally:
kill_process_tree(process.pid)
```

Error:
```
metrics = run_eval(args)
> self.assertGreaterEqual(metrics["score"], 0.65)
E AssertionError: 0.0625 not greater than or equal to 0.65

test/srt/test_wave_attention_backend.py:59: AssertionError
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.