antirez / antirez/ds4

Proposal: serve token_embd.weight row lookups from the host mapping (~1 GiB residency win on streaming setups)

オープン
#688 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
C
スター
22.3k
フォーク
2.1k
平均マージ
1日 3時間
マージ済み PR(30日)
4

説明

### Observation

`token_embd.weight` is accessed one row (~8-16 KB) per decoded token, but it is prepared as a fully resident device span at startup -- ~1 GiB on DeepSeek V4 Flash Q8_0. On streaming setups (the configurations `--ssd-streaming` exists for), it is the single largest always-resident tensor, and its access pattern -- one read-only row per token -- is exactly what a read-only mmap serves at page-cache speed.

### Proposal

Under an env gate (`DS4_EMBD_MMAP=1`, default OFF): leave `token_embd.weight` out of the resident device spans at startup, and have the embed path resolve it to the read-only host model mapping instead. Hot rows stay in page cache; per-token decode touches one row.

### Measurements (GB10, DeepSeek V4 Flash Q8_0, streaming, ctx 131072)

| measure | resident (baseline) | DS4_EMBD_MMAP=1 |
|---|---|---|
| residency | 0.99 GiB device spans | 0.00 GiB -- **-1010 MiB resident** |
| warm decode e2e t/s (3 turns) | 4.25 / 4.28 / 4.36 | 4.19 / 4.23 / 4.33 (-0.9% avg, within run-to-run noise) |
| cold prefill (3430 tok) | baseline | no cost signal |
| worst case: first token after a full page-cache drop | -- | one-time ~2.5 s transient (4.8 s vs 2.3 s warm), recovered by the next request |

Verdict on that box: ~1 GiB of residency freed for no measurable throughput cost.

### Important caveat

The measurement and the patch rely on **unified-memory ATS/HMM pageable access** (GB10 / Grace class): the GPU reads the file-backed host pages directly. On discrete-GPU targets the same idea would need the `cudaHostRegister`'d-mapping path that `cuda_model_range_ptr` already has -- untested by me, and honestly future work. The env gate defaults OFF precisely so non-unified targets are unaffected.

PR with the env-gated implementation attached.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。