antirez / antirez/ds4

Metal wires entire mmap-backed model views on first GPU use (ssd-streaming, mixed-size expert layers)

Ouverte
#638 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
C
Étoiles
22.3k
Forks
2.1k
Merge moyen
1 j 3 h
PR mergées (30 j)
4

Description

On mixed-precision MoE models, layers whose experts fall outside the slab size class bypass the expert cache and read experts through mmap-backed model views wrapped in Metal buffers.

We measured (M4 Max 128 GB, macOS 26, DeepSeek 4 Flash: 6/43 such layers, 20.25 GiB of views) that the driver wires the **entire mapped views** at first decode use — 65.2 GiB wired vs 3.4 at rest — regardless of how few bytes each token actually touches. On 48 GB machines this makes any honest expert-cache budget impossible; at 64 GB it still consumes a third of the budget.

Isolated reproduction (~200-line C+Metal benchmark, same file, same 2 GiB of scattered reads under a GPU kernel):

- explicit 2 MiB-aligned slot buffers filled by `pread` and wrapped with `newBufferWithBytesNoCopy`: wired growth **≤ 0.8 GiB**;
- a single bytesNoCopy buffer over an 8 GiB mmap of the same file: wired growth **+8–10 GiB** for the same 2 GiB touched.

Verdict was identical across interleaved repeat pairs; checksums CPU=GPU=across-arms all runs. Happy to share the benchmark source.

Two independent public runtimes avoid this class of problem by never mapping the expert pool: TurboFieldfare (Swift/Metal, bounded slot pool + parallel `pread`) and WASTE (CPU, `pread` + 4 KiB records).

If there is interest, we can propose an opt-in slot-based read path for bypass-class layers, or contribute the measurement evidence to a design discussion. Related: the honest-measurement mode in #637 is what made the wired behavior visible in the first place.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.