sgl-project / sgl-project/sglang
[Bug] GLM-5.3-Flash crashes at startup under pipeline parallelism — KeyError: 'residual'
- Dominant language
- Python
- Stars
- 36.1k
- Forks
- 9k
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 239
Description
### Checklist
- [x] I searched related issues but found no solution.
- [x] The bug persists in the latest version.
- [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
- [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
- [x] Please use English. Otherwise, it will be closed.
### Describe the bug
GLM-5.3-Flash (glm5_next) fails during warmup with pp_size > 1. All weights load successfully; the crash happens at the first forward pass, in the FlashInfer autotune dummy run, on every non-first pipeline stage.
The cause is a disagreement about whether an mHC model carries residual as a separate PP proxy tensor:
srt/model_executor/runner/base_runner.py (warmup buffer allocation) assumes mHC models fold residual into hidden_states, and therefore does not allocate a residual buffer at all:
### Reproduction
docker run --gpus all --ipc=host --shm-size 32g \
-v /path/to/GLM-5.3-Flash-NVFP4:/model:ro \
lmsysorg/sglang:glm-5.3-flash \
python3 -m sglang.launch_server \
--model-path /model \
--pp-size 3 --tp-size 1 \
--context-length 32768 \
--mem-fraction-static 0.90 \
--kv-cache-dtype fp8_e4m3 \
--disable-shared-experts-fusion \
--trust-remote-code \
--host 0.0.0.0 --port 30000
Result:
File "sglang/srt/model_executor/runner/base_runner.py", line 316, in forward_fn
self._dummy_run(
File "sglang/srt/model_executor/runner/base_runner.py", line 669, in _dummy_run
run_once()
File "sglang/srt/model_executor/runner/base_runner.py", line 657, in run_once
logits_output_or_pp_proxy_tensors = mr.model.forward(
File "sglang/srt/models/glm5_next.py", line 1529, in forward
hidden_states = general_mm_embed_routine(
File "sglang/srt/managers/mm_utils.py", line 730, in general_mm_embed_routine
hidden_states = language_model(
File "sglang/srt/models/glm5_next.py", line 1093, in forward
residual = pp_proxy_tensors["residual"]
~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "sglang/srt/model_executor/forward_batch_info.py", line 1778, in __getitem__
return self.tensors[key]
~~~~~~~~~~~~^^^^^
KeyError: 'residual'
### Environment
sglang image : lmsysorg/sglang:glm-5.3-flash
sglang : 0.0.0.dev1+g033446bb05
model : LibertAIDAI/GLM-5.3-Flash-NVFP4 (ModelOpt 0.45.0, rev 11d73216cd)
GPUs : 3 x NVIDIA RTX PRO 6000 Blackwell Max-Q, 97887 MiB each (SM120)
driver : 610.57.04
topology : --pp-size 3 --tp-size 1
attention : attention_backend=dsa, dsa_prefill_backend=trtllm, dsa_decode_backend=trtllm
Contributor guide
Research direction
Start with the warmup buffer allocation in srt/model_executor/runner/base_runner.py and the pipeline-stage forward path in srt/models/glm5_next.py, using the provided pp-size 3 reproduction. Trace how pp_proxy_tensors are created and read during the FlashInfer dummy run; done means startup completes without the residual KeyError on non-first pipeline stages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- ai, backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100