antirez / antirez/ds4

Metal wires entire mmap-backed model views on first GPU use (ssd-streaming, mixed-size expert layers)

Aperta
#638 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
22.3k
Fork
2.1k
Merge medio
1g 3h
PR unite (30g)
4

Descrizione

On mixed-precision MoE models, layers whose experts fall outside the slab size class bypass the expert cache and read experts through mmap-backed model views wrapped in Metal buffers.

We measured (M4 Max 128 GB, macOS 26, DeepSeek 4 Flash: 6/43 such layers, 20.25 GiB of views) that the driver wires the **entire mapped views** at first decode use — 65.2 GiB wired vs 3.4 at rest — regardless of how few bytes each token actually touches. On 48 GB machines this makes any honest expert-cache budget impossible; at 64 GB it still consumes a third of the budget.

Isolated reproduction (~200-line C+Metal benchmark, same file, same 2 GiB of scattered reads under a GPU kernel):

- explicit 2 MiB-aligned slot buffers filled by `pread` and wrapped with `newBufferWithBytesNoCopy`: wired growth **≤ 0.8 GiB**;
- a single bytesNoCopy buffer over an 8 GiB mmap of the same file: wired growth **+8–10 GiB** for the same 2 GiB touched.

Verdict was identical across interleaved repeat pairs; checksums CPU=GPU=across-arms all runs. Happy to share the benchmark source.

Two independent public runtimes avoid this class of problem by never mapping the expert pool: TurboFieldfare (Swift/Metal, bounded slot pool + parallel `pread`) and WASTE (CPU, `pread` + 4 KiB records).

If there is interest, we can propose an opt-in slot-based read path for bypass-class layers, or contribute the measurement evidence to a design discussion. Related: the honest-measurement mode in #637 is what made the wired behavior visible in the first place.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.