antirez / antirez/ds4

Metal wires entire mmap-backed model views on first GPU use (ssd-streaming, mixed-size expert layers)

未关闭
#638 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C
星标
22.3k
派生
2.1k
平均合并
1 天 3 小时
30 天内合并 PR
4

描述

On mixed-precision MoE models, layers whose experts fall outside the slab size class bypass the expert cache and read experts through mmap-backed model views wrapped in Metal buffers.

We measured (M4 Max 128 GB, macOS 26, DeepSeek 4 Flash: 6/43 such layers, 20.25 GiB of views) that the driver wires the **entire mapped views** at first decode use — 65.2 GiB wired vs 3.4 at rest — regardless of how few bytes each token actually touches. On 48 GB machines this makes any honest expert-cache budget impossible; at 64 GB it still consumes a third of the budget.

Isolated reproduction (~200-line C+Metal benchmark, same file, same 2 GiB of scattered reads under a GPU kernel):

- explicit 2 MiB-aligned slot buffers filled by `pread` and wrapped with `newBufferWithBytesNoCopy`: wired growth **≤ 0.8 GiB**;
- a single bytesNoCopy buffer over an 8 GiB mmap of the same file: wired growth **+8–10 GiB** for the same 2 GiB touched.

Verdict was identical across interleaved repeat pairs; checksums CPU=GPU=across-arms all runs. Happy to share the benchmark source.

Two independent public runtimes avoid this class of problem by never mapping the expert pool: TurboFieldfare (Swift/Metal, bounded slot pool + parallel `pread`) and WASTE (CPU, `pread` + 4 KiB records).

If there is interest, we can propose an opt-in slot-based read path for bypass-class layers, or contribute the measurement evidence to a design discussion. Related: the honest-measurement mode in #637 is what made the wired behavior visible in the first place.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。