sgl-project / sgl-project/sglang

[Bug] GLM-5.3-Flash crashes at startup under pipeline parallelism — KeyError: 'residual'

Open
#36,906 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
36.1k
Forks
9k
Avg merge
1d 5h
Merged PRs (30d)
239

Description

### Checklist

- [x] I searched related issues but found no solution.
- [x] The bug persists in the latest version.
- [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
- [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
- [x] Please use English. Otherwise, it will be closed.

### Describe the bug

GLM-5.3-Flash (glm5_next) fails during warmup with pp_size > 1. All weights load successfully; the crash happens at the first forward pass, in the FlashInfer autotune dummy run, on every non-first pipeline stage.

The cause is a disagreement about whether an mHC model carries residual as a separate PP proxy tensor:

srt/model_executor/runner/base_runner.py (warmup buffer allocation) assumes mHC models fold residual into hidden_states, and therefore does not allocate a residual buffer at all:

### Reproduction

docker run --gpus all --ipc=host --shm-size 32g \
-v /path/to/GLM-5.3-Flash-NVFP4:/model:ro \
lmsysorg/sglang:glm-5.3-flash \
python3 -m sglang.launch_server \
--model-path /model \
--pp-size 3 --tp-size 1 \
--context-length 32768 \
--mem-fraction-static 0.90 \
--kv-cache-dtype fp8_e4m3 \
--disable-shared-experts-fusion \
--trust-remote-code \
--host 0.0.0.0 --port 30000

Result:

File "sglang/srt/model_executor/runner/base_runner.py", line 316, in forward_fn
self._dummy_run(
File "sglang/srt/model_executor/runner/base_runner.py", line 669, in _dummy_run
run_once()
File "sglang/srt/model_executor/runner/base_runner.py", line 657, in run_once
logits_output_or_pp_proxy_tensors = mr.model.forward(
File "sglang/srt/models/glm5_next.py", line 1529, in forward
hidden_states = general_mm_embed_routine(
File "sglang/srt/managers/mm_utils.py", line 730, in general_mm_embed_routine
hidden_states = language_model(
File "sglang/srt/models/glm5_next.py", line 1093, in forward
residual = pp_proxy_tensors["residual"]
~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "sglang/srt/model_executor/forward_batch_info.py", line 1778, in __getitem__
return self.tensors[key]
~~~~~~~~~~~~^^^^^
KeyError: 'residual'

### Environment

sglang image : lmsysorg/sglang:glm-5.3-flash
sglang : 0.0.0.dev1+g033446bb05
model : LibertAIDAI/GLM-5.3-Flash-NVFP4 (ModelOpt 0.45.0, rev 11d73216cd)
GPUs : 3 x NVIDIA RTX PRO 6000 Blackwell Max-Q, 97887 MiB each (SM120)
driver : 610.57.04
topology : --pp-size 3 --tp-size 1
attention : attention_backend=dsa, dsa_prefill_backend=trtllm, dsa_decode_backend=trtllm

Contributor guide

Open the contributing guide

Research direction

Start with the warmup buffer allocation in srt/model_executor/runner/base_runner.py and the pipeline-stage forward path in srt/models/glm5_next.py, using the provided pp-size 3 reproduction. Trace how pp_proxy_tensors are created and read during the FlashInfer dummy run; done means startup completes without the residual KeyError on non-first pipeline stages.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
ai, backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.