deepseek-ai / deepseek-ai/DeepEP

JIT cache: rename-race loser crashes with `EP_HOST_ASSERT(runtime != nullptr)` when the cache is on a shared filesystem (NFS)

Open
#683 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

## Summary

`Compiler::build()` handles concurrent builds of the same kernel by compiling into a private temp directory and atomically renaming it into the cache; a losing process deletes its temp directory and re-reads the winner's (`csrc/jit/compiler.hpp:156-158`):

```cpp
const auto runtime = kernel_runtime_cache->get(dir_path);
EP_HOST_ASSERT(runtime != nullptr);
```

This assumes a failed rename implies the winner's directory is visible to the loser. On a local filesystem that is guaranteed; on NFS it is not, and that is the default configuration, since the cache falls back to `$HOME/.deep_ep` (`compiler.hpp:52`) and home directories are commonly NFS on multi-node clusters. The losing rank dies on the assert and kills the job. Observed on 2.0.0 (EPv2); code unchanged on current `main` (60d44037).

## Mechanism

Cold start, all ranks race `build()` for the same kernel signature:

1. Every rank misses at `compiler.hpp:116`. that stat primes each node's NFS client *negative* lookup cache for `dir_path` ("does not exist"), valid until the parent directory's attribute cache expires (up to ~60 s on default mounts).
2. All ranks compile in parallel into private `/tmp/` dirs; the first finisher's rename lands atomically at the server.
3. A loser on another node gets an authoritative rename failure, deletes its temp dir, and re-stats `dir_path`. Answered "does not exist" from its still-valid negative cache entry, without a server round-trip. `check_validity` → false, `get()` → `nullptr`, assert at `compiler.hpp:158`.

The existing fsync/atomic-rename hardening is writer-side (durability and ordering at the server); this failure is reader-side, in the loser's own client cache, which writer-side fsync cannot invalidate. A failed `rename()` does not invalidate the caller's cached lookup of the target either.

Reproduced on the first attempt with 16 ranks across two 8-GPU H200 nodes and a cold NFS cache: all 8 ranks of the losing node crashed at the assert while JIT-building the dispatch/combine kernels (whole-node failure as the stale cache belongs to the node's NFS client), and the winner node's ranks hung in the next collective until the job was killed.

GB200/GB300-class systems (4 GPUs per node) are more exposed: every EP ≥ 8 job spans multiple NFS clients, so a shared cache races from the smallest multi-node shape (a GB300 deployment we checked shares an NFSv4.2 filesystem with default caching options). The interconnect is irrelevant — the race is in the host-side compile path, identical on aarch64.

## Proposed fix: #684

The loser does not need the winner's directory: it holds freshly compiled, equivalent artifacts (the cache key hashes kernel name, source, flags, and compiler signature). Load the runtime from its own temp directory, cache it in-memory under the canonical key, then delete the temp dir. This is safe because `KernelRuntime`'s constructor consumes the files eagerly (`cuobjdump` + `cudaLibraryLoadFromFile`/`cuModuleLoad`) and keeps only GPU handles; warm starts still read the shared cache as before, and other rename failures degrade gracefully instead of asserting.

Validated on the reproducing setup: the crashing scenario passes 16/16 with the fix, with the losing branch exercised 73 times across 80 rank×kernel builds (verified via `EP_JIT_DEBUG=1` load paths); a warm-start rerun serves all 80 loads from the canonical cache directories.

## Notes

- Workaround until fixed: point `EP_JIT_CACHE_DIR` at a node-local path (e.g. under `/tmp`).
- A per-kernel `flock()` (cf. deepseek-ai/DeepGEMM#301) would reduce the N-way duplicate compilation but not fix this crash: acquiring a lock does not invalidate the client's cached lookup of an unrelated path, and `flock` is not cross-node on `nolock` NFSv3 mounts.
- With `EP_JIT_DUMP_*` set, the losing rank's dumps are deleted with its temp dir under the fix; the winner's remain in the cache.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.