deepseek-ai / deepseek-ai/DeepGEMM
DeepGEMM JIT: CUDA 800 (NOT_PERMITTED) at jit/handle.hpp:154 kills vLLM engine on first long-context request
- Dominant language
- Cuda
- Stars
- 7.8k
- Forks
- 1.3k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 3
Description
## Environment
GB10 (DGX Spark, sm_121a), CUDA 13.2, driver 595.x
vLLM local build v0.25.2.dev0+g752a3a504.d20260714, TP=2, CUDA graphs FULL_AND_PIECEWISE (lazy capture), chunked prefill + prefix caching, MTP-6 spec decode
Model: DeepSeek-V4-Flash (hybrid mamba+MLA+sparse indexer MoE), fp8 weights, nvfp4_ds_mla KV
DeepGEMM: vendored, offline-nvcc JIT (DG_JIT_USE_NVRTC=0), persistent cache, PDL enabled (set_pdl(True))
## Error (first decode step after a 287k-token prefill):
```
RuntimeError: CUDA driver error
(/workspace/.deps/deepgemm-src/csrc/apis/../jit_kernels/impls/../../jit/handle.hpp:154):
800 (CUDA_ERROR_NOT_PERMITTED, operation not permitted)
→ EngineDeadError → API server exit
```
## Call path
deepseek_v4/attention.py → flashinfer_sparse.py:606 _o_proj → ops/o_proj.py fp8_einsum("bhr,hdr->bhd") → DeepGEMM JIT
### Trigger
First-ever execution of the sparse o_proj einsum path: a ~287k-token request (2 concurrent, 124k/125k prefix-cache hits) after 29 h uptime with only short-context traffic. Prefill itself succeeded (286,720 tokens computed, KV 10% used). Boot-time DeepGEMM warmup (1,762 kernels) does not cover this path/shape.
## Suspected cause
The nvcc compile step succeeds; the failure is at the driver call in handle.hpp:154 (module load or PDL-attributed launch). JIT module load / PDL launch is prohibited while any stream in the context is capturing — and vLLM lazily captures a new decode graph exactly when a new batch shape appears, so a first-time JIT compile can land inside a capture window.
- Error 800 = operation not permitted (not OOM, not invalid handle)
- Intermittent by nature: after restart, six long-context runs (64k–509k tokens, incl. 2-concurrent 255k) all passed; the compiled shape is then cached and never hits the JIT path again
## Question
Which driver call is at jit/handle.hpp:154 in the current jit_kernels/impls layout — module load or PDL launch? And is there a recommended capture-safe pattern (e.g., defer load while cudaStreamIsCapturing, or retry-once-on-800)?
## Proposed fix
Defer cuModuleLoadData until capture ends (queue + load after cudaStreamEndCapture), or retry once on 800. A set_pdl(False) toggle would also be a useful operator workaround. Happy to test a patched build and provide the full crash log.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at jit/handle.hpp:154 and trace the current jit_kernels/impls layout to identify whether the failing driver call is module loading or a PDL launch. Reproduce the first-time sparse o_proj shape during lazy CUDA graph capture using the provided environment and call path. Done means the failing call is established and a capture-safe handling pattern or set_pdl(False) workaround is verified.
Written by the indexing model from the issue text.
Assessment
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100