iree-org / iree-org/wave

[SGLang] Llama3.1 model crashes with cuda graphs

Open
#466 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
59
Forks
32
PR merge metrics
No merged PRs in 30d

Description

The following command crashes with a GPU memory error.
` python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend wave`

Error:
`Memory access fault by GPU node-2 (Agent handle: 0x5a0d7202eab0) on address 0x7d35a0c00000. Reason: Unknown.`

This doesn't seem to crash inside the kernel but rather outside after a specific number of iterations. Cause of the crash is due to corrupt kernel inputs for example `query` tensor for the decode kernel contains `nan` values.

Contributor guide

Open the contributing guide

Research direction

Start by running the provided `sglang.launch_server` command with the Llama-3.1-8B-Instruct model and `--attention-backend wave`, then observe when the GPU memory error occurs. The report points to corrupt kernel inputs, including NaN values in the decode kernel's `query` tensor; done means the command no longer crashes after repeated iterations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.