[SGLang] Llama3.1 model crashes with cuda graphs
- Dominant language
- Python
- Stars
- 59
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
The following command crashes with a GPU memory error.
` python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend wave`
Error:
`Memory access fault by GPU node-2 (Agent handle: 0x5a0d7202eab0) on address 0x7d35a0c00000. Reason: Unknown.`
This doesn't seem to crash inside the kernel but rather outside after a specific number of iterations. Cause of the crash is due to corrupt kernel inputs for example `query` tensor for the decode kernel contains `nan` values.
Contributor guide
Research direction
Start by running the provided `sglang.launch_server` command with the Llama-3.1-8B-Instruct model and `--attention-backend wave`, then observe when the GPU memory error occurs. The report points to corrupt kernel inputs, including NaN values in the decode kernel's `query` tensor; done means the command no longer crashes after repeated iterations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100