antirez / antirez/ds4

Metal wires entire mmap-backed model views on first GPU use (ssd-streaming, mixed-size expert layers)

Open
#638 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

On mixed-precision MoE models, layers whose experts fall outside the slab size class bypass the expert cache and read experts through mmap-backed model views wrapped in Metal buffers.

We measured (M4 Max 128 GB, macOS 26, DeepSeek 4 Flash: 6/43 such layers, 20.25 GiB of views) that the driver wires the **entire mapped views** at first decode use — 65.2 GiB wired vs 3.4 at rest — regardless of how few bytes each token actually touches. On 48 GB machines this makes any honest expert-cache budget impossible; at 64 GB it still consumes a third of the budget.

Isolated reproduction (~200-line C+Metal benchmark, same file, same 2 GiB of scattered reads under a GPU kernel):

- explicit 2 MiB-aligned slot buffers filled by `pread` and wrapped with `newBufferWithBytesNoCopy`: wired growth **≤ 0.8 GiB**;
- a single bytesNoCopy buffer over an 8 GiB mmap of the same file: wired growth **+8–10 GiB** for the same 2 GiB touched.

Verdict was identical across interleaved repeat pairs; checksums CPU=GPU=across-arms all runs. Happy to share the benchmark source.

Two independent public runtimes avoid this class of problem by never mapping the expert pool: TurboFieldfare (Swift/Metal, bounded slot pool + parallel `pread`) and WASTE (CPU, `pread` + 4 KiB records).

If there is interest, we can propose an opt-in slot-based read path for bypass-class layers, or contribute the measurement evidence to a design discussion. Related: the honest-measurement mode in #637 is what made the wired behavior visible in the first place.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.