lablup / lablup/mlxcel

perf(bench): Confirm or clear three small-model prefill drops at 0.6.0

Open
#1,614 2 comments 0 reactions 0 assignees View on GitHub
area:benchmark priority:low status:ready type:performance
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

## Problem / Background

The 0.6.0 M5 Max sweep (`benchmarks/metal_m5max_2026-09-03.csv`) against 0.4.0-rc.1 (`benchmarks/metal_m5max_2026-07-12.csv`) has zero decode regressions across 159 comparable checkpoints and 13 prefill drops over 10%. Ten are resolved in `docs/benchmark_results/model_tests_m5max.md:385`: their prompt length changed, and since `prefill_tok_s` is prompt tokens over prefill ms, a shorter prompt amortizes fixed per-call overhead over fewer tokens and reads lower. Three dropped at an **unchanged** prompt length, so that explanation does not cover them.

## Current Behavior

| Checkpoint | prefill 07-12 | prefill 09-03 | delta | prompt tokens |
|---|---|---|---|---|
| `dots.ocr-4bit` | 834.77 | 630.78 | -24.4% | 10 in both |
| `smollm-135m-4bit` | 6093.98 | 5105.84 | -16.2% | 16 in both |
| `stablelm-1.6b-4bit` | 2883.43 | 2487.59 | -13.7% | 26 in both |

This is an investigation, not a regression report. Each figure is one sample from one sweep, at 10 to 26 prompt tokens where per-call overhead dominates, and two subjects are tiny models where this project's benchmark noise is known to be large. `dots.ocr-4bit` is also flagged partial at `model_tests_m5max.md:278` (loads and prefills, emits zero text tokens on a text-only prompt), so its numbers are the least trustworthy.

## Proposed Solution

Re-measure each subject with a repeat count, paired against a control at the identical prompt length whose prefill did not move across the same sweep pair. Controls read from the same two CSVs:

| Subject (prompt tokens) | Control | Control 07-12 -> 09-03 | Delta |
|---|---|---|---|
| `dots.ocr-4bit` (10) | `phi-3-mini-4bit` | 585.51 -> 591.88 | +1.1% |
| `smollm-135m-4bit` (16) | `gemma3-1b-4bit` | 2070.99 -> 1938.99 | -6.4% |
| `stablelm-1.6b-4bit` (26) | `qwen2.5-1.5b-4bit` | 3356.34 -> 3105.70 | -7.5% |

The control's repeat spread defines the noise band. A drop that survives it is a change that moves the numbers: take a teacher-forced logit trace at the prefill width per `docs/benchmarks.md:263` rather than arguing from throughput. Do not git-bisect 0.4.0-rc.1..0.6.0 first; that is expensive while the effect may not exist.

## Scope

**In scope:** the three subjects plus three controls on M5 Max, and the outcome recorded in `docs/benchmark_results/model_tests_m5max.md`.

**Out of scope:** the ten prompt-length drops (already resolved), decode (clean in this pair), the `dots.ocr-4bit` no-text defect itself, and M1 Ultra or CUDA re-measurement.

## Implementation Notes

- **Reuse** the single-model path, `./scripts/bench_decode.sh --cooldown 30 --big-cooldown 30 models/`. No new harness; all six model directories already exist under `models/`.
- **Same-day repeats overwrite**: `scripts/bench_decode.sh:589` truncates via `echo "$CSV_HEADER" > "$OUTPUT"` and auto-names single runs `metal_m5max__single_.csv`. Give each repeat a distinct `--suffix` (yielding `metal_m5max___single_.csv`) or an explicit `--output`.
- **Single runs start colder than sweep rows**: global pre-warm and settle only runs in `all` mode (`scripts/bench_decode.sh:626`), though each run still does its own `--warmup-tokens` pass. Measure each control in the same session as its subject; do not compare a fresh single run against the sweep CSV.
- **Read at a longer prompt too**: repeat with `--prompt-tokens 512` so the result is not dominated by per-call overhead. Report alongside the short-prompt number, not instead of it.
- **Edge cases**: `dots.ocr-4bit` records `decode_tok_s=0.00` in both sweeps, so only prefill is usable and no decode delta should be reported. A row classified `FAIL:*` or `SKIP:*` is not a sample; re-run it. Overlapping spreads means cleared, not inconclusive.

## Acceptance Criteria

- [ ] Each subject has at least 5 prefill samples at its original prompt length with min, median, and max reported, and its control measured in the same session at the same repeat count.
- [ ] Each of the three is classified as a confirmed regression (drop survives the control's spread, cause identified) or cleared as noise.
- [ ] Any confirmed regression carries a teacher-forced logit trace at the prefill width per `docs/benchmarks.md:263`.
- [ ] The paragraph at `docs/benchmark_results/model_tests_m5max.md:446` is replaced with the outcome, so the next sweep does not re-open this.
- [ ] Raw re-measurement CSVs are committed under `benchmarks/`.

## Verification

```bash
cargo build --release --features metal,accelerate

# One subject plus its control, 5 repeats, distinct suffix per repeat.
for i in 1 2 3 4 5; do
./scripts/bench_decode.sh --cooldown 30 --big-cooldown 30 --suffix "rep$i" models/smollm-135m-4bit
./scripts/bench_decode.sh --cooldown 30 --big-cooldown 30 --suffix "rep$i" models/gemma3-1b-4bit
done
# Repeat for dots.ocr-4bit/phi-3-mini-4bit and stablelm-1.6b-4bit/qwen2.5-1.5b-4bit,
# then again per pair with --prompt-tokens 512 --suffix "long$i".

awk -F, 'FNR>1 {print FILENAME"\t"$1"\t"$3"\t"$6}' benchmarks/metal_m5max_*_rep*_single_*.csv
```

A pass is a recorded classification for all three, not a throughput target: each subject's spread either overlaps its control's (cleared) or sits clearly below it (confirmed, then traced).

## Technical Considerations

Follow-up to the 0.6.0 M5 Max campaign. Methodologically mirrors #755, which re-measured the 2026-07-12 GB10 drops before attributing them. Low priority: decode is clean sweep-wide, and the worst-affected checkpoint produces no usable text anyway.

Contributor guide

Open the contributing guide

Research direction

Start with docs/benchmarks.md:263, docs/benchmark_results/model_tests_m5max.md:446, and scripts/bench_decode.sh:589 and :626; build the release binary and run paired repeats for the three subjects and controls on M5 Max. Use the specified short and 512-token prompts, excluding FAIL or SKIP rows, then compare each subject's spread with its control. Done means classifying all three, adding any required trace, committing raw CSVs under benchmarks/, and replacing the results paragraph.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, shell
Domain
documentation, performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.