RFC: benchmark-driven default ASR model selection (CPU + GPU)

Summary

Choose the default Copy Assist model by measuring the host's actual inference throughput and picking the largest tier that can keep up with real-time speech, instead of the current binary GPU-yes/no guess. Fixes the under-served low end (old CPU-only laptops/desktops — a large slice of the ham community) and the naive high end (a weak integrated GPU assumed capable of large-v3-turbo).

Motivation

Today (AsrModelCatalog::defaultTierId()):

if (gpuBackendAvailable()) return "large-v3-turbo";  // ANY gpu → 1.6 GB model
return "base";                                        // ANY cpu → 141 MB model

Two problems, symmetric:

• Low end: every CPU-only host gets base, regardless of CPU capability. Measured on an RPi5 (used in dev as a deliberate proxy for "old ham laptop"): with a correct release build, even the tiny model is only borderline real-time, and base cannot keep up. So the default is a model the hardware can't run in real-time. A QA tester on a very old Windows laptop is hitting exactly this.
• High end: any GPU → large-v3-turbo. A weak/old integrated GPU may not sustain turbo in real-time either, but it's assumed to.

The catalog has no notion of how fast the device actually is.

Proposal

A small synthetic benchmark, run once (cached), that selects the default tier from measured throughput. Two stages so we never block on a large download:

Stage 1 — instant, model-free CPU pre-pick

Use the built-in whisper_bench_ggml_mul_mat() (~1–2 s, no model download). It reports GFLOPS per quant type — including Q8_0 and F16, which are exactly the tiny/base formats. Map GFLOPS → a coarse CPU tier via a calibration table. This alone is enough to stop an old laptop from ever starting on base.

Stage 2 — refined, real-inference RTF probe (CPU and GPU)

Download the smallest tier once (tiny-q8_0, ~42 MB — becomes a usable model anyway), run a fixed synthetic clip through it on the active backend, and measure the real RTF (real-time factor = decode_time ÷ audio_duration). Extrapolate to every tier using per-tier compute ratios, and pick the largest tier with projected RTF ≤ 0.5 (headroom for VAD/resample/burst load and the fact that decode can't start until a segment closes). This path is backend-agnostic — it's how GPU gets measured, since the mul_mat bench is CPU-only and GPUs need a real graph to measure meaningfully.

Tier mapping

A calibration table (RTF thresholds / GFLOPS floors per tier), seeded from reference devices (RPi5 as the low-end anchor, a mid laptop, a discrete-GPU box) rather than hard-coded guesses. Bands: ≤0.5 comfortable → eligible; 0.5–0.8 marginal → allowed with a "may lag under load" note; >0.8 → step down.

Add quantized-tiny tiers

Add tiny-q8_0 (~42 MB) and tiny-q5_1 to AsrModelCatalog as first-class downloadable tiers — the natural low-end defaults, and the Stage-2 probe model.

UX

• Runs on first ASR enable, not every launch. Result cached, keyed to a hardware signature (CPU model + core count + GPU id) so it re-runs only when hardware changes.
• Shows the recommendation + measured RTF ("Benchmarked your hardware → recommending Tiny (q8_0), ~0.4× real-time"). User override always wins — this sets the default, not a lock.
• A "Re-run benchmark" button in Copy Assist settings.

Non-goals / out of scope

• Not changing the inference engine or segmenter.
• Not auto-switching models mid-session based on live backlog (possible follow-up; this RFC is about the initial default).
• Model accuracy per language/tier is separate (see the language-selector work).

Open questions

1. Does whisper_bench_ggml_mul_mat's GFLOPS correlate tightly enough with real whisper RTF to skip Stage 2 on CPU, or is the Stage-2 probe always worth it?
2. Calibration-table maintenance: ship seeded values, or crowd-source RTF telemetry (opt-in)?
3. GPU probe: acceptable to require the one-time 42 MB tiny-q8_0 download to benchmark a GPU host, or add a GPU-specific microbench?
4. Where to draw the marginal-band cutoff (0.5 vs 0.6)?