antirez / antirez/ds4

--ssd-streaming-cache-experts is silently inert on CUDA — expert cache never populated, hit rate 0

Open
#639 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

On the CUDA backend, the `--ssd-streaming-cache-experts` budget has no effect: `ds4_gpu_set_streaming_expert_cache_budget()` / `_expert_bytes()` are no-op stubs (`(void)bytes;`) and `ds4_gpu_stream_expert_cache_configured_count()` hard-returns 0. `cuda_stream_selected_cache_begin_load()` unconditionally invalidates and re-fetches every selected expert's gate/up/down bytes from the mmap'd file on every token — the flag parses, prints its budget, and does nothing. The Metal backend has a real per-(layer,expert) LRU (`g_stream_expert_cache`, `..._peek` / `_install_loaded` / `_prune_global`), so this looks like the CUDA half never landed rather than a design choice.

Measured on a GB10 (DGX Spark class, sm_121), DeepSeek-V4-Flash, `--ssd-streaming`:

- decode tok/s is flat (~1.0) across cache budgets 8GB-100GB, cold vs popularity-preload identical
- instrumented fetch counters show hit rate exactly 0.000 and ~3GB/token read from file at every budget
- NVMe sustained at ~85% of its ceiling while GPU sits at 18% SM — fully disk-bound on re-fetch traffic

We ported the Metal LRU design to CUDA on our research fork (device-resident entries keyed (layer,expert), global LRU prune, budget semantics per the CLI contract): decode went 1.03 -> 2.96 tok/s at 100GB budget with 81% measured hit rate, greedy outputs byte-identical. Branch with the port + measurements: https://github.com/nexus-cw/ds4/tree/research/gb10 (LRU port in 416533c; counters in 1961967; numbers in research/gb10/MEASUREMENTS.md). Happy to shape it into a PR if the approach fits how you'd want the CUDA side to work.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.