chore: bench_decode.sh all measures duplicate checkpoints twice

> **Body rewritten 2026-09-04.** The body originally filed here was a copy of #1616 (batched MoE decode scaling) and did not describe this issue at all; the two differed only in cosmetic wording. Everything below is reconstructed from the script and from the committed benchmark CSVs. The title was correct.

## Problem / Background

`bench_decode.sh all` enumerates `"$MODELS_DIR"/*/` and benchmarks every directory it finds (`scripts/bench_decode.sh:658` and `:666`). Nothing deduplicates, so a checkpoint present under two directory names is loaded, prefilled, decoded and cooled down twice, and lands in the CSV as two rows under two names. Downstream the duplicate rows are treated as two distinct models.

## Evidence

Scanning `models/mlx` on M1 Ultra by `sha256(config.json)` plus total `*.safetensors` bytes finds **15 duplicate groups covering 16 redundant directories out of 199**, holding **124.5 GB** of redundant weights.

| Group | Copies | Weights each |
|---|---|---|
| `hunyuan-13b` / `hunyuan-a13b-instruct-4bit` | 2 | 45.22 GB |
| `nemotron-h-30b-4bit` / `nemotron-nas-30b-4bit` | 2 | 17.78 GB |
| `qwen3-30b-a3b-4bit` / `qwen3-moe-4bit` | 2 | 17.17 GB |
| `qwen2.5-vl-3b-hf` / `qwen2.5-vl-3b-instruct` | 2 | 7.51 GB |
| `pixtral-12b` / `pixtral-12b-4bit` | 2 | 7.14 GB |
| `qwen3-vl-8b-4bit` / `qwen3-vl-8b-instruct-4bit` | 2 | 5.76 GB |
| `llama-3.1-8b-4bit` / `meta-llama-3.1-8b-instruct-4bit` | 2 | 4.52 GB |
| `qwen2.5-7b` / `qwen2.5-7b-4bit` / `qwen2.5-7b-instruct-4bit` | 3 | 4.28 GB |
| `gemma-3-4b-it-4bit` / `gemma3-4b-4bit` | 2 | 3.40 GB |
| `qwen3-vl-4b-4bit` / `qwen3-vl-4b-instruct-4bit` | 2 | 3.09 GB |
| `qwen3-vl-2b` / `qwen3-vl-2b-4bit` | 2 | 1.78 GB |
| `qwen2-vl-2b` / `qwen2-vl-2b-4bit` | 2 | 1.25 GB |
| `gemma-3-1b-it-4bit` / `gemma3-1b-4bit` | 2 | 0.73 GB |
| `qwen3-0.6b` / `qwen3-0.6b-4bit` | 2 | 0.34 GB |
| `qwen2-0.5b` / `qwen2.5-0.5b-4bit` | 2 | 0.28 GB |

Six groups were additionally spot-checked by shard count plus a sha256 over the first and last 1 MiB of every shard: all six match, and none share inodes, so these are independent full copies rather than hardlinks or APFS clones. Names are misleading in both directions: `pixtral-12b` and `pixtral-12b-4bit` are the same 4-bit checkpoint, and `hunyuan-13b` is the same file set as `hunyuan-a13b-instruct-4bit`.

**The duplication is in every committed sweep, not just on this host.** Counting group members that co-occur in each committed CSV:

| Sweep | Rows | Redundant rows |
|---|---|---|
| `metal_m1ultra_2026-07-12.csv` | 168 | 15 |
| `metal_m1ultra_2026-07-06.csv` | 169 | 14 |
| `metal_m1ultra_2026-06-15.csv` | 151 | 13 |
| `metal_m5max_2026-07-12.csv` | 176 | 14 |
| `metal_m5max_2026-06-15.csv` | 151 | 13 |
| `metal_m5max_2026-04-04.csv` | 95 | 2 |

The paired numbers agree to within run noise, which is itself confirmation that the two rows measure one checkpoint: `qwen2.5-7b` / `qwen2.5-7b-4bit` / `qwen2.5-7b-instruct-4bit` decode at 108.85 / 108.47 / 108.86 tok/s in `metal_m1ultra_2026-07-12.csv`, and `nemotron-h-30b-4bit` / `nemotron-nas-30b-4bit` at 91.75 / 90.37. The VLM sweeps carry the same redundancy at the same counts.

**It reaches published documentation.** Three duplicate groups appear as separate table rows in each of `docs/benchmark_results/model_tests_m1ultra.md` and `docs/benchmark_results/model_tests_m5max.md` (`gemma-3-4b-it-4bit` / `gemma3-4b-4bit`, `nemotron-h-30b-4bit` / `nemotron-nas-30b-4bit`, `pixtral-12b` / `pixtral-12b-4bit`, `qwen3-30b-a3b-4bit` / `qwen3-moe-4bit` depending on the file), so one checkpoint is presented as two supported models. The parity comparison tables in the same documents list only one member per group, which means the deduplication currently exists only as unrecorded manual curation between the CSV and the document. That is not reproducible and it silently drifts every time someone regenerates a table.

## Current Behavior

```bash
if [[ "$MODEL_ARG" == "all" ]]; then
# First pass: run all models except known GPU-crash models
for dir in "$MODELS_DIR"/*/; do
[[ -d "$dir" ]] || continue
is_gpu_crash_model "$dir" && continue
result=$(bench_one "$dir")
```

The two-pass split at `:658` and `:666` exists to defer `GPU_CRASH_MODELS` (`:603`) to the end; each directory is still visited exactly once per sweep. The defect is that two directories holding one checkpoint are two directories. `bench_one` (`:414`) keys everything on the directory basename, and `emit` (`:595`) appends whatever it returns, so the CSV inherits the directory naming with no identity notion at all.

## Proposed Solution

Deduplicate by checkpoint identity in `all` mode, before the first pass, and keep single-model mode untouched so an explicit path always measures exactly what was named.

- Compute a cheap identity key per candidate directory: `sha256(config.json)` plus the sorted list of `(*.safetensors basename, size)`. This costs one small read plus a `stat` per shard, needs no weight hashing, and correctly separates checkpoints that differ only in quantization (their `config.json` differs in the `quantization` block).
- Also collapse directories that resolve to the same path, so a symlink such as `models/mlx/gemma-3-1b-it-4bit` into the shared model store never doubles a row.
- Measure one member per group and skip the rest, emitting one row per skipped alias with a `SKIP:duplicate_of=` status so the CSV stays self-describing and a reader can see the alias set rather than silently wondering where a model went. Row count then stays stable across sweeps, which matters because `docs/benchmark_results/model_tests.md` indexes sweeps by measured row count. The survivor only has to be deterministic; as implemented it is the first directory the sweep's own glob reaches, which means the longer name wins for a prefix pair (`pixtral-12b-4bit` over `pixtral-12b`).
- Print the collapsed groups to stderr at sweep start so the operator sees what was folded before a multi-hour run begins.
- Add `--no-dedup` to restore today's behavior for anyone who deliberately wants to measure two copies, for example when checking whether a re-download is bit-identical.

Rejected: deleting the redundant directories. `models/` is gitignored local state, the aliases are how various scripts and documents already refer to these checkpoints (`scripts/bench_embeddings.py`, the deferred-pairing tables), and a sweep must not be silently wrong on a host that happens to hold two copies. The harness has to be correct regardless of how the model store is arranged.

## Scope

**In scope:** `scripts/bench_decode.sh` (`all` mode enumeration, the new dedup helper, `--no-dedup`, usage text at `:285`), and a note in `docs/benchmarks.md` describing the identity key and the `SKIP:duplicate_of` status.

**Out of scope:** deleting or renaming any directory under `models/`; regenerating past sweeps; correcting the duplicate rows already present in committed CSVs and in `model_tests_m1ultra.md` and `model_tests_m5max.md`, which is a documentation pass that should follow once sweeps stop producing them; `scripts/bench_embeddings.py` and `scripts/bench_serving_concurrency.py`, which take explicit model lists rather than enumerating a directory.

## Acceptance Criteria

- [ ] `bench_decode.sh all` measures each distinct checkpoint exactly once, with the alias names emitted as `SKIP:duplicate_of=` rows so total row count is unchanged.
- [ ] The identity key does not collapse checkpoints that differ only in quantization or in weight dtype; a test or a documented check demonstrates this on a real pair from `models/mlx`, for example `bitnet-b1.58-2b-4t` against `bitnet-b1.58-2b-4t-4bit`.
- [ ] A directory that is a symlink to another measured directory is collapsed as well.
- [ ] `--no-dedup` reproduces the current behavior exactly.
- [ ] Single-model mode (`bench_decode.sh models/`) is unaffected and still measures the named path.
- [ ] The collapsed groups are printed to stderr before the first measurement.
- [ ] `docs/benchmarks.md` documents the identity key and the new status string.

## Verification

```bash
# Sanity-check the script and see the grouping the sweep would apply. The
# collapsed groups print to stderr before the first model is measured, so a
# sweep against a small temporary MODELS_DIR shows them without a long run.
bash -n scripts/bench_decode.sh

# Cheap end-to-end check: build a temporary MODELS_DIR of symlinks holding two
# small duplicate groups (gemma-3-1b-it-4bit / gemma3-1b-4bit and qwen2-0.5b /
# qwen2.5-0.5b-4bit) plus one non-duplicate control, then run both arms.
MODELS_DIR= ./scripts/bench_decode.sh all --output /tmp/dedup.csv
MODELS_DIR= ./scripts/bench_decode.sh all --no-dedup --output /tmp/nodedup.csv
```

A pass is: one measured row per distinct checkpoint, the alias rows present and labelled, the sweep's total row count equal to the directory count so the CSV index in `docs/benchmark_results/model_tests.md` stays meaningful, and `--no-dedup` measuring every directory. There is no `--dry-run`; an earlier draft of this body referenced one that was never implemented, and the stderr group report plus a temporary `MODELS_DIR` cover the same need.

## Technical Considerations

Relevant to #1614, which compares per-checkpoint prefill across two sweeps: duplicate rows make a "checkpoint" ambiguous in exactly the comparison that issue performs. Filed alongside #1612, #1613, #1614 and #1616 from the same 2026-09-04 benchmark campaign.

Start with scripts/bench_decode.sh usage text at :285 and the enumeration, bench_one, emit, and GPU-crash handling at :414, :595, :603, :658, and :666. Run bash -n, then use the temporary MODELS_DIR verification commands to compare normal and --no-dedup sweeps. Done means distinct checkpoints are measured once, aliases emit SKIP:duplicate_of rows, symlinks and quantization differences are handled, and docs/benchmarks.md describes the behavior.

Written by the indexing model from the issue text.