Laguna server batch prefill fails at >=7k tokens: routed MoE intermediate quantize launch invalid argument
- Vorherrschende Sprache
- C
- Sterne
- 22.3k
- Forks
- 2.1k
- Ø Merge
- 1 T. 3 Std.
- Gemergte PRs (30 T.)
- 4
Beschreibung
## Environment
- Hardware: NVIDIA GB10 (DGX Spark class, sm_121a), CUDA 13, Linux aarch64
- Branch: `laguna-s2.1` @ `448d569` ("Tune Laguna sampling defaults")
- Model: Laguna-S-2.1 Q4_K_M GGUF (~65 GiB), fully resident, `--cuda`
- Server: `ds4-server -m laguna-s-2.1-Q4_K_M.gguf --cuda --ctx 32768`
## Symptom
Any chat/messages request whose prompt is >= ~7,069 tokens fails server-side prefill immediately after the first layer:
```
ds4: CUDA Laguna routed MoE intermediate quantize launch failed: invalid argument
ds4: Laguna batch prefill failed in routed experts after 1/48 layers
```
The client gets a 500; every retry fails identically (deterministic, not transient).
## Bisection (server chat path, ctx 32768)
| prompt tokens | result |
|---|---|
| <= 6,049 | prefill OK (e.g. `6049/6049 avg=558.56 t/s 10.830s`, generation fine) |
| >= ~7,069 | fails as above, always after layer 1/48 |
| 24,438 (real workload) | fails, every retry |
## Why this looks like launch geometry, not memory
- Plenty of free memory at failure time (~58+ GiB available); the same server handles short prompts fine at the same ctx.
- The **one-shot CLI path** (`ds4` binary, same model, same build) prefilled a ~16k-token prompt without error at ~2,198 t/s — the failure is specific to the **server batch prefill** path.
- `invalid argument` from a CUDA launch at a token-count threshold in the low-7k range is the classic signature of a grid-dimension limit being exceeded (e.g. tokens x top-k experts overflowing a 65,535-capped grid dimension) in the batched routed-MoE intermediate-quantize launch. Offered as a hypothesis — we have not bisected the kernel itself.
## No mitigation available
`--prefill-chunk` is a hard startup reject for Laguna, so prompts cannot be chunked below the threshold.
## Minimal repro
1. Build `laguna-s2.1` @ 448d569 with CUDA; start `ds4-server -m --cuda --ctx 32768 --port 8010`.
2. Send one OpenAI chat completion with a ~7,500-token prompt:
```bash
python3 -c 'import json,urllib.request,random; random.seed(1); w=["alpha","bravo","charlie","delta","echo","foxtrot","golf","hotel","india","juliet"]; p="Summarize this list briefly: "+" ".join(random.choice(w) for _ in range(7500)); body=json.dumps({"model":"laguna-s-2.1-chat","messages":[{"role":"user","content":p}],"max_tokens":20}).encode(); print(urllib.request.urlopen(urllib.request.Request("http://127.0.0.1:8010/v1/chat/completions",data=body,headers={"Content-Type":"application/json"})).read())'
```
Expected: completion. Actual: HTTP 500 with the two log lines above; a short prompt on the same running server returns 200.
## Real-workload impact
We hit this running the claude CLI against the server's Anthropic-compatible surface: its ~24.4k-token first request can never complete (8 retries, clean API error), so any long-system-prompt agent client is hard-blocked on Laguna.
Re-verified today (2026-08-05) at the same branch head before filing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.