Empirical RSS curve for incremental ctx loading (10K to 250K) on M4 Max 128GB - is --kv-disk-dir masking the indexer growth?

Hey Salvatore,

First — thank you for `ds4-server`. I've been running DeepSeek V4-Flash on an M4 Max 128 GB and your Metal+CUDA implementation is the cleanest path I've seen. This question comes from real-world benchmarking, not theoretical concern.

## TL;DR

I ran an **incremental context-loading test** (10K → 250K tokens, single session, 10K increments, 25 datapoints) on the 2-bit quant (`ds4flash.gguf`, 81 GB — your Q2-K, the one your README recommends for 128 GB Macs). The README states:

> "Full context of 1M tokens is going to use more or less 26GB of memory (compressed indexer alone will be like 22GB)"

Linearly extrapolated, this predicts **~22 KB/token** of compressed-indexer growth. My measurement on the M4 Max shows the **actual incremental RSS growth is ~66× smaller** than that linear prediction (effective ~0.33 KB/token vs the implied 22 KB/token), and the curve is essentially flat — even mildly decreasing — across the 10K→250K range. I want to understand whether (a) my measurement methodology is flawed, (b) the README's 22 GB figure includes `--kv-disk-dir`-backed pages that don't surface in process RSS, or (c) the indexer has a pool-reuse / batch-allocation pattern that makes per-token linear extrapolation misleading.

## Setup

- **Hardware**: Apple M4 Max, 128 GB unified memory, macOS 15.x
- **Build**: `ds4-server` built from `ds4.c` source on 2026-05-15 (a commit prior to `d0357ec1` of 2026-05-15T21:18Z; my local working tree's inner `.git` was empty so I can't pin the exact SHA — happy to re-run on any specific commit you'd like to anchor on).
- **Model**: `ds4flash.gguf` (the 2-bit Q2-K quant, 81 GB on disk — what your README recommends for 128 GB Macs)
- **Startup**: `ds4-server --ctx 300000 --port 8080 --model ds4flash.gguf --warm-weights`
- **No** `--kv-disk-dir` set (pure in-memory mode)
- **Measurement**: process RSS via `psutil.Process(server_pid).memory_info().rss`, sampled immediately after each incremental forward pass completes. Server PID captured at startup, so the measurement targets `ds4-server` specifically (not the test harness process — I had an earlier methodology bug measuring the wrong process; the data below is from the corrected run).
- **Loading method**: append a fixed-size text block to the running session every 10K tokens, force a 1-token decode after each append so the server fully ingests the new context, then sample RSS

## Data (25 datapoints, 10K → 250K)

| ctx_tokens | RSS (GB) | ΔRSS from 10K (MB) | prefill (s) |
|-----------:|---------:|-------------------:|------------:|
| 10,000 | 86.769 | 0 | 48.7 |
| 20,000 | 86.776 | +6.5 | 103.7 |
| 30,000 | 86.786 | +16.6 | 164.2 |
| 40,000 | 86.787 | +17.2 | 222.1 |
| 50,000 | 86.787 | +17.9 | 283.0 |
| 60,000 | 86.809 | +39.5 | 347.3 |
| 70,000 | 86.810 | +40.3 | 414.0 |
| 80,000 | 86.812 | +43.0 | 483.6 |
| 90,000 | 86.829 | +59.9 | 567.9 |
| 100,000 | 86.833 | +64.0 | 637.2 |
| 110,000 | 86.830 | +61.0 | 721.2 |
| 120,000 | 86.870 | +100.2 | 797.9 |
| 130,000 | 86.872 | +102.9 | 885.4 |
| 140,000 | 86.881 | +111.9 | 969.1 |
| 150,000 | 86.906 | +137.0 | 1063.0 |
| 160,000 | 86.912 | +142.9 | 1150.5 |
| 170,000 | 86.910 | +140.5 | 1251.4 |
| 180,000 | 86.959 | +189.6 | 1362.6 |
| 190,000 | 86.962 | +192.7 | 1448.1 |
| 200,000 | 86.971 | +201.4 | 1539.9 |
| 210,000 | 86.963 | +193.6 | 1641.7 |
| 220,000 | 86.968 | +199.7 | 1764.7 |
| 230,000 | 86.963 | +193.7 | 1860.3 |
| 240,000 | 86.953 | +183.4 | 1982.6 |
| **250,000**| **86.846**| **+78.5** | **2110.2** |

Raw JSONL + reproducer script + chart: https://github.com/michaelhuo2030/millisecond-era (under `data/` and `scripts/`).

**Notable: RSS at 250K (86.846 GB) is *lower* than at 200K (86.971 GB)** — the indexer pool appears to actively reclaim memory at high ctx rather than monotonically grow.

## Observation

Over a 240K-token context delta (10K → 250K), **process RSS grew by 78.5 MB**. Linearly extrapolating your README's "22 GB @ 1M" figure to 240K tokens predicts **~5.16 GB of indexer growth**.

**Measured / linear-predicted ≈ 1.5%** (**98.5% suppression**).
**Effective ~0.33 KB/token measured** vs README implies ~22 KB/token (**~66× over-estimate**).

If I extend the linear curve from my data to 1M tokens, I get an indexer projection of **~330 MB** — not 22 GB.

## Three hypotheses I'm holding

**Hypothesis A — Pool reuse / allocator amortization** (currently my leading hypothesis)
The indexer pre-allocates a large pool at ctx-startup (which is why RSS at 10K is already ~86.77 GB, ~6 GB above the 81 GB Q2-K GGUF on disk) and subsequent increments reuse the pool until it fills. The 10K→250K window appears to sit entirely inside one pool epoch — and the slight RSS *decrease* at 250K (86.846 GB vs 200K at 86.971 GB) suggests active reclaim, not just static pre-allocation.

**Hypothesis B — `--kv-disk-dir`-backed indexer pages**
The "22 GB @ 1M" figure in the README may count disk-backed indexer pages (via `--kv-disk-dir` or anonymous mmap with swap backing), which don't show in process RSS. With `--kv-disk-dir` unset, perhaps the indexer is silently spilling to swap and the OS reports only resident pages.

I checked `vm_stat` during the run — swap-ins / swap-outs were quiet (no significant paging), but I'm not 100% sure my methodology catches this if the indexer uses purgeable / compressed memory regions.

**Hypothesis C — README is a worst-case bound, not a typical**
Possible the 22 GB figure was measured on a stress prompt (e.g., adversarial tokens that maximize hash collisions in the indexer) and typical workloads see much smaller growth.

## What I'd love your read on

1. Which hypothesis (A / B / C / something else) matches your actual implementation?
2. Is there an instrumentation hook in `ds4-server` to print indexer-specific memory (rather than total process RSS)? I'm happy to recompile with a flag and rerun — I've actually prototyped a small `#ifdef DS4_MEMLOG_ENABLED` patch that emits per-category alloc events to stderr; happy to clean it up as a PR if useful.
3. If the README's 22 GB figure is a known-stress / worst-case number, would you accept a PR that adds a sentence clarifying that typical workloads see roughly an order of magnitude less?

## Context (why I care)

I'm doing 28nm ReRAM-CIM edge inference chip research for DeepSeek V4-Flash class models. The indexer's true memory footprint is **load-bearing** for whether 300K context can fit on a single die. The difference between "22 KB/token linear" and "1 KB/token plateaued" is the difference between "needs 14nm hybrid + chiplet" and "single 28nm die works."

If you're curious about the bigger picture, I've put up a public thesis at **https://github.com/michaelhuo2030/millisecond-era** with all 50+ datapoints reproducible, the design open, the budget honest (~¥10K out of pocket, 1.5 months, one person), and a transparent list of what's verified vs what's still assumed. Not pitching you on anything — just sharing where this work is grounded.

If you'd find it useful, I'm happy to:
- Share the full harness Python script (~280 lines, runs in ~6 hours end-to-end on M4 Max)
- Run additional experiments (multi-user concurrency, cold disk-cache, OOM frontier) you'd want to see
- Co-author a more rigorous benchmark write-up

One more thing — when the first chip prototype actually exists, the first unit goes to you. Not for anything in return, not for marketing, just because that's where this whole thing started. The 22 KB/token number in your README is precisely what made the substrate-layer thesis I'd been holding *empirically falsifiable* the night your code ran on our laptop. The least we can do is hand back the first physical artifact when it exists.

Thanks again for `ds4-server` — it's been the kind of code that makes me want to actually understand what's happening underneath.

— Michael (@michaelhuo2030)