deepseek-ai / deepseek-ai/DeepGEMM

DeepGEMM JIT: CUDA 800 (NOT_PERMITTED) at jit/handle.hpp:154 kills vLLM engine on first long-context request

Open
#443 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
7.8k
Forks
1.3k
Avg merge
3d 7h
Merged PRs (30d)
3

Description

## Environment

GB10 (DGX Spark, sm_121a), CUDA 13.2, driver 595.x
vLLM local build v0.25.2.dev0+g752a3a504.d20260714, TP=2, CUDA graphs FULL_AND_PIECEWISE (lazy capture), chunked prefill + prefix caching, MTP-6 spec decode
Model: DeepSeek-V4-Flash (hybrid mamba+MLA+sparse indexer MoE), fp8 weights, nvfp4_ds_mla KV
DeepGEMM: vendored, offline-nvcc JIT (DG_JIT_USE_NVRTC=0), persistent cache, PDL enabled (set_pdl(True))

## Error (first decode step after a 287k-token prefill):
```
RuntimeError: CUDA driver error
(/workspace/.deps/deepgemm-src/csrc/apis/../jit_kernels/impls/../../jit/handle.hpp:154):
800 (CUDA_ERROR_NOT_PERMITTED, operation not permitted)
→ EngineDeadError → API server exit
```

## Call path
deepseek_v4/attention.py → flashinfer_sparse.py:606 _o_proj → ops/o_proj.py fp8_einsum("bhr,hdr->bhd") → DeepGEMM JIT

### Trigger
First-ever execution of the sparse o_proj einsum path: a ~287k-token request (2 concurrent, 124k/125k prefix-cache hits) after 29 h uptime with only short-context traffic. Prefill itself succeeded (286,720 tokens computed, KV 10% used). Boot-time DeepGEMM warmup (1,762 kernels) does not cover this path/shape.

## Suspected cause
The nvcc compile step succeeds; the failure is at the driver call in handle.hpp:154 (module load or PDL-attributed launch). JIT module load / PDL launch is prohibited while any stream in the context is capturing — and vLLM lazily captures a new decode graph exactly when a new batch shape appears, so a first-time JIT compile can land inside a capture window.

- Error 800 = operation not permitted (not OOM, not invalid handle)
- Intermittent by nature: after restart, six long-context runs (64k–509k tokens, incl. 2-concurrent 255k) all passed; the compiled shape is then cached and never hits the JIT path again

## Question
Which driver call is at jit/handle.hpp:154 in the current jit_kernels/impls layout — module load or PDL launch? And is there a recommended capture-safe pattern (e.g., defer load while cudaStreamIsCapturing, or retry-once-on-800)?

## Proposed fix
Defer cuModuleLoadData until capture ends (queue + load after cudaStreamEndCapture), or retry once on 800. A set_pdl(False) toggle would also be a useful operator workaround. Happy to test a patched build and provide the full crash log.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at jit/handle.hpp:154 and trace the current jit_kernels/impls layout to identify whether the failing driver call is module loading or a PDL launch. Reproduce the first-time sparse o_proj shape during lazy CUDA graph capture using the provided environment and call path. Done means the failing call is established and a capture-safe handling pattern or set_pdl(False) workaround is verified.

Written by the indexing model from the issue text.

Assessment

Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.