sgl-project / sgl-project/sglang

[GB10] Long prefill (>40k tokens) exhausts unified memory and silently kills the worker rank — no traceback, no OOM record; cross-stack control passes at 54k

Open
#36,941 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
36.1k
Forks
9k
Avg merge
1d 5h
Merged PRs (30d)
239

Description

## Summary
On 2x DGX Spark (GB10, sm_121, TP=2, 121.7 GB unified memory per node), GLM-5.3-Flash prefills above roughly 40k tokens drive **host/unified memory on the worker node to its ceiling**, and somewhere between 54k and 62k tokens the worker rank dies with no traceback, no CUDA error, and no OOM record. The head then hangs on a collective and self-terminates.

This is chunked prefill (`--chunked-prefill-size 8192`), so per-step token counts are bounded — but measured memory grows with **total prompt length**, not chunk size, which suggests something in the prefill path retains per-token state across chunks.

## Measured
Config: SGLang @ `xinyuan/glm-5.3-flash-support` (aa8c950a3) + our fp8-KV patches (#36904), DFLASH speculative (D=5), `--kv-cache-dtype fp8_e4m3`, `--mem-fraction-static 0.92`, `--max-running-requests 8`, `--max-mamba-cache-size 40 --mamba-ssm-dtype bfloat16`, context 131072, KV pool 244,288 tokens.

Single request, needle-in-haystack prompt, `enable_thinking: false`, `max_tokens: 100`:

| prompt tokens | result | worker host memory (spark-2, of 124.6 GB) |
|---|---|---|
| ~32k | PASS (recall correct) | not sampled |
| ~40k | PASS but ~10 min wall (client timed out at 600 s; server kept generating) | **121.3 GB peak** |
| ~62k (on a 262144-context variant, fraction 0.93) | **worker container gone**, head hung then killed itself | not sampled |

Head-side symptom at the death:
```
[rank1] ProcessGroupNCCL ... Received a dump signal due to a collective timeout from rank 1
[TP1] Scheduler hit an exception: RuntimeError: ... gloo/transport/tcp/pair.cc:537
Read error [10.10.10.1]:51085: Connection reset by peer
Received sigquit from a child process.
```
Docker reports `OOMKilled=false ExitCode=0 FinishedAt=0001-01-01T00:00:00Z` for the dead worker — i.e. the container's own state record is incoherent, consistent with the process being reaped outside docker's accounting rather than exiting.

Once the request is aborted and containers are removed, memory returns immediately (121.3 GB → 4.5 GB), so this is live allocation during prefill, not a leak that persists.

## Cross-stack control on the SAME hardware, same night
We ran a different serving stack on these exact two nodes (vLLM-based, EXL3 4bpw weights, fp8 KV, DFlash2 drafter, MLA/drafter KV page-sharing): identical needle prompts at **34,840 and 54,040 tokens both PASSED**, worker alive, peak host memory 119.3 GB. So the hardware and the fabric sustain 54k-token prefills; the failure is specific to this serving path.

## Why this may matter beyond our rig
Silent worker death on long prefill has been reported independently on GB10 with a different quantization (NVFP4 KV, ~6k tokens) and was assumed to be quant-specific. Our data falsifies that scoping: it reproduces on fp8_e4m3 KV at a higher threshold, so prompt length looks like the governing variable, not KV dtype.

## Asks
1. Is any prefill-path buffer expected to scale with **total** sequence length rather than chunk size on the DSA/tilelang path (indexer top-k workspaces, sparse index buffers, radix bookkeeping)?
2. Would a lower `--mem-fraction-static` plus smaller `--chunked-prefill-size` be the supported mitigation, or is there a bound we should be setting explicitly? (We are testing 2048/0.85 tonight and will post the result either way.)
3. On unified-memory parts, is there an intended guard that fails the request instead of the process when host memory is exhausted? A request-level error would turn a node kill into a retry.

Happy to run instrumented probes — we have the rig, the harness, and both stacks side by side.

Env: 2x DGX Spark GB10 (sm_121, aarch64), CUDA 13, TP=2/nnodes=2, dual-rail CX-7 RoCE, SGLang @ aa8c950a3 + PR #36904. Recipe/ladder: https://github.com/beastllama/GLM-5.3-Flash-DFlash2-SGLang-2x-DGX-Spark
Related: #36889 (mamba cap), #36904 (fp8 KV), #36548 (DFLASH concurrency correctness on GB10).

Contributor guide

Open the contributing guide

Research direction

Start with the DSA/tilelang prefill path and reproduce the prompt-length ladder from the linked recipe on the stated GB10 setup. Compare memory growth across chunk sizes and inspect the related issues #36889, #36904, and #36548 for relevant entry points. Done means isolating the retained allocation or failure boundary and documenting a validated request-level failure or supported mitigation.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.