--ssd-streaming-cache-experts is silently inert on CUDA — expert cache never populated, hit rate 0
- Dominant language
- C
- Stars
- 22.3k
- Forks
- 2.1k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
On the CUDA backend, the `--ssd-streaming-cache-experts` budget has no effect: `ds4_gpu_set_streaming_expert_cache_budget()` / `_expert_bytes()` are no-op stubs (`(void)bytes;`) and `ds4_gpu_stream_expert_cache_configured_count()` hard-returns 0. `cuda_stream_selected_cache_begin_load()` unconditionally invalidates and re-fetches every selected expert's gate/up/down bytes from the mmap'd file on every token — the flag parses, prints its budget, and does nothing. The Metal backend has a real per-(layer,expert) LRU (`g_stream_expert_cache`, `..._peek` / `_install_loaded` / `_prune_global`), so this looks like the CUDA half never landed rather than a design choice.
Measured on a GB10 (DGX Spark class, sm_121), DeepSeek-V4-Flash, `--ssd-streaming`:
- decode tok/s is flat (~1.0) across cache budgets 8GB-100GB, cold vs popularity-preload identical
- instrumented fetch counters show hit rate exactly 0.000 and ~3GB/token read from file at every budget
- NVMe sustained at ~85% of its ceiling while GPU sits at 18% SM — fully disk-bound on re-fetch traffic
We ported the Metal LRU design to CUDA on our research fork (device-resident entries keyed (layer,expert), global LRU prune, budget semantics per the CLI contract): decode went 1.03 -> 2.96 tok/s at 100GB budget with 81% measured hit rate, greedy outputs byte-identical. Branch with the port + measurements: https://github.com/nexus-cw/ds4/tree/research/gb10 (LRU port in 416533c; counters in 1961967; numbers in research/gb10/MEASUREMENTS.md). Happy to shape it into a PR if the approach fits how you'd want the CUDA side to work.
Contributor guide
Assessment
This issue has not been assessed yet.