Validate SSIMULACRA2 as the non-photo oracle: single-observer human study on the imazen-26 non-photo corpus

## Motivation — ssim2 is the non-photo "oracle" we have never validated against humans

zensim's evaluation has a `nonphoto` corpus — **"imazen-26 non-photo (held-out)"**, defined in
[`zensim-validate/src/bin/bake_verdict.rs`](https://github.com/imazen/zensim/blob/main/zensim-validate/src/bin/bake_verdict.rs)
(the `Corpus { name: "nonphoto", … }` block). Its target column is called `human_score`, but the
comment in that same block is explicit that the value is **`ssim2/100`**:

> `human_score = ssim2/100, so SROCC here = rank-agreement with ssim2 on diverse content.`

So the non-photo gate's "oracle" is **SSIMULACRA2, not human opinion.** This is deliberate: every
human-MOS dataset we own (CID22, KADID, TID, KonJND, AIC-3/4) is photographic, and there is a
standing user directive that *"ssim2 is the best north star for non-photo content"* (screen/UI,
rendered text/documents/code, line-art/vector/icons/diagrams/comics, charts/plots/maps, AI-gen,
game/3D renders, pixel-art). ssim2 is used as a **proxy oracle** for all of it.

**The open question this issue exists to answer: is ssim2 actually a good oracle for non-photo
content?** We have never checked. If ssim2 is merely weak on, say, line-art, the gate is noisy; if
it is *inverted* on a category, then the entire `nonphoto` gate — and any model that ranks well on
it (the `G-NP` gate, the dashboard composite that weights `0.30·nonphoto`) — is optimizing toward
the wrong target on that category, invisibly. Context on how central and how unvalidated this is:
zensim `benchmarks/stats_correctness_review_2026-07-26.md` (Part 4 + the `|SROCC|`-hides-inversion
finding in §2d: a globally inverted metric currently reads as a *high positive* headline).

[squintly](https://github.com/imazen/squintly) is our psychovisual pairwise-collection tool, and it
is the right instrument for a first-pass human check. This issue scopes a **single-observer, ~5-hour
sanity validation** and enumerates the concrete squintly work it needs.

---

## What squintly already is (so we build on it, not around it)

squintly is **substantial** — this is not a blank slate. Verified against the current tree:

- **2AFC pairwise + tie is the default trial type** and the analysis backbone: BT-Davidson-with-ties
scale reconstruction ([`src/bt.rs`](src/bt.rs) — `fit`, `with_monotonicity`, `beta_to_quality`),
ASAP expected-information-gain active sampling ([`src/asap.rs`](src/asap.rs) — `eig`, `pick_max_eig`),
wired into pair selection via [`src/sampling.rs`](src/sampling.rs) (`select_pair_with_eig`,
`ASAP_MIN_OBS = 8`). README §3 lever table: #1 (ASAP) ✅, #12 (unified BT+ACR) ✅.
- **Per-trial viewing-condition capture, staircase thresholds, honeypots, session grading A–F →
`session_weight ∈ {0,0.5,1.0,1.5}`, qualifier, engagement/auth** — all ✅.
- **A curator + variant pipeline** ([`src/curator.rs`](src/curator.rs) — `Candidate` struct at
`curator.rs:42`: `sha256 / corpus / relative_path / suspected_category / license_id / blob_url /
source_q_detected / …`) and **four TSV exports** ([`src/export.rs`](src/export.rs), routes at
[`src/main.rs`](src/main.rs) `:259–262`): `pareto` (per-encoding BT-Davidson θ + bootstrap CI),
`thresholds`, `responses`, `unified`.
- A **pre-registered study** ([`docs/STUDY.md`](docs/STUDY.md)) — but that study is a **different
study** from this one. It is a phone-first, anonymous, **crowd** collection producing zensim-**v48
training labels** on web-**photo** content, conditioned on viewing conditions; anchors are meant to
come "from CID22 source overlap"; outputs are BT θ + threshold functions *for training*. Phase
status: **v0.2 finishing** (🟡), pilot not yet run.

This issue asks for a small, **read-mostly reuse** of that machinery for a *validation* purpose it
was not built for. The reuse is high-leverage; the gaps below are real and specific.

---

## Study design

**Protocol: pairwise 2AFC, not star-ratings.** With a single observer, forced-choice is far more
reliable than ACR ratings and is exactly what a rank-agreement (SROCC) test needs. squintly already
defaults to 2AFC + ASAP + BT-Davidson, so this is the native path.

**Throughput.** At ~4–6 s per judgment, 5 hours yields **~2,000–2,500 quality judgments**, split
into **~5 × 45-min sessions** (single-observer quality collapses after ~1 h — the sessions must be
short and spaced). With ASAP surfacing the highest-EIG pairs (~2–4 comparisons per condition), that
budget reconstructs within-reference human scales over **~40–60 references × ~8–12 conditions ≈
400–600 conditions** (~1,600–2,400 comparisons).

**Confidence-interval math (include it, because it's what makes 5 hours decisive).** Pooling the
reconstructed human-θ-vs-ssim2 rank agreement across ~400 conditions and using the Fisher-z SE
`1/√(N−3)`:

- At **N = 300, ρ = 0.85**: z = atanh(0.85) = 1.256, SE_z = 1/√297 = 0.058 → 95% CI in z
[1.142, 1.370] → **SROCC 95% CI ≈ [0.815, 0.879]** (±~0.03).
- Past **N ≈ 100** the "ssim2 works (~0.85)" and "ssim2 struggles (~0.70)" hypotheses **separate**:
at N=100 their 95% CIs are ≈[0.78, 0.90] vs ≈[0.58, 0.79] — barely non-overlapping, and the gap
widens fast with N.

So the **pooled** number is decisive for "is ssim2 broadly OK on non-photo." The **per-reference**
resolution (N = 40–60 refs) is coarser but is precisely what surfaces an **inverted or collapsed
category** — the failure mode we most need to catch.

**Analysis output.** Reconstruct human θ per reference (BT-Davidson within-reference, reusing
`bt::fit` + `beta_to_quality` exactly as `export::pareto_tsv` already does — each reference is
self-anchored at θ_reference = 0), **join each condition's θ with that condition's ssim2**, and emit
**per-reference and per-category human-vs-ssim2 SROCC / PLCC with bootstrap CI**, plus the pooled
number. Within-reference is the correct granularity: no cross-reference scale alignment is needed
(which matters here because non-photo has no CID22 anchors to align to).

---

## Corpus

Validate the **real gate's oracle**, not a proxy for it: draw a **stratified ~40–60-reference
subsample of the same imazen-26 non-photo content the `nonphoto` gate scores.**

Source pool (verified): the gate reads
`nonphoto_features_372col_2026-07-15.parquet`
(`/mnt/v/zen/zensim-training/2026-05-15-full-features/…`). Measured: **10,000 condition rows across
1,518 unique references** (`ref_basename`, e.g. `o_1011.png.scale108x192` — an imazen-26 origin at a
resize scale), `human_score ∈ [0,1]` = ssim2/100. **Important:** this parquet is a *derived feature
table* — it carries only `ref_basename + human_score + f0..f371`. It has **no codec, no q, no raw
ssim2, no image path**. So building a viewable, human-judgeable corpus requires a rejoin (work item
#1 below): map each chosen `ref_basename` back to the **bigcodec / imazen-26 canonical**
(`s3://zentrain/ext720-canonical-2026-07-22/bigcodec/`; reference images under `/mnt/v/imazen-26/`;
see `~/work/zen/DATA_PROVENANCE.md`) to recover, per condition, the **(codec, q, ssim2, encoded
distorted-variant URL, pristine reference URL)** and the **category label**.

Per reference, take its **existing real-codec conditions** (zenjpeg / zenavif / zenjxl / zenwebp × 3–4
q spanning the quality range ≈ **8–12 conditions/ref**) with the **ssim2 already computed for each**.
Stratify the ~40–60 references across categories (target counts; exact counts documented at assembly):

| Category group | Target refs |
|---|---|
| Screenshots / UI / dashboards | 8–10 |
| Rendered text / documents / code | 6–8 |
| Line-art / vector / icons / diagrams / comics | 8–10 |
| Charts / plots / maps | 5–7 |
| AI-generated | 5–7 |
| Game / 3D-render | 4–6 |
| Pixel-art / low-color | 3–5 |

The line-art / rendered-text / pixel-art rows are the ones most likely to expose an ssim2 failure, so
weight the stratification toward them rather than toward the photo-adjacent "AI-gen / 3D" rows.

---

## Work needed (this is the "lot of work" — checklist, each item points at real code)

- [ ] **1. Non-photo corpus ingestion — the main lift.** Assemble the ~40–60-ref manifest: (a)
stratified-subsample `ref_basename`s from the nonphoto pool across the category table above;
(b) **rejoin each `ref_basename` to the bigcodec/imazen-26 canonical** to recover per-condition
`(codec, q, ssim2, distorted-variant URL, reference URL)` + category; (c) land the category into
curator's `suspected_category`; (d) ingest through the candidate flow
([`src/curator.rs`](src/curator.rs) — `Candidate` at `curator.rs:42`, via the manifest-TSV /
JSON-candidate parse + `upsert`). The images must be **reachable** by the browser — either
mirror the reference + distorted variants into the coefficient store squintly consumes, or point
each `blob_url` at the bigcodec R2 variant URLs. This is the bulk of the effort.

- [ ] **2. ssim2 as a first-class condition metric + a human-vs-metric validation export.** Grep
confirms **zero** `ssim2` references anywhere in `src/` (only in `docs/`), and **no
metric-vs-human correlation exists** — the four exports ([`src/export.rs`](src/export.rs)) emit
only BT-Davidson θ. Need: (a) carry a **per-condition ssim2** value from ingestion to a joinable
place (a new column on the encoding/candidate metadata, or a sidecar keyed on encoding-id /
`(ref_basename, codec, q)`); (b) a **new analysis + export** — e.g. `/api/export/validation.tsv`
(route in [`src/main.rs`](src/main.rs), handler in [`src/handlers.rs`](src/handlers.rs), impl in
[`src/export.rs`](src/export.rs)) — that reconstructs per-reference human θ (reuse `bt::fit` +
`beta_to_quality`, as `pareto_tsv` does) and joins each condition's ssim2, emitting **per-ref +
per-category SROCC / PLCC + bootstrap CI**. **Stats must come from the canonical owner** —
zensim's `zenstats` (shell it) or `scripts/lib/zen_stats.py` — **not hand-rolled.** squintly's
[`src/stats.rs`](src/stats.rs) has bootstrap/ci90 but no SROCC, and per zensim's no-duplication
rule the correlation stats belong to zenstats.

- [ ] **3. Control the metric-coupling in scale reconstruction (corrected — this is *not* just
"disable the trivial filter").** Verified: the current trivial-pair filter
([`src/sampling.rs:350`](src/sampling.rs) `is_trivial_pair`) is a **bytes-ratio (>4×) /
quality-gap (>30) heuristic, not a metric ensemble**, and `try_pair` (`sampling.rs:223`) samples
only **adjacent same-codec** pairs — which *skip*, they don't inject, so they don't poison the
scale. The metric-ensemble predictive-fill is **PS-PC (lever #13, 🔵 not yet wired)** — keep it
**off** for validation (turning it on would inject metric-predicted outcomes → circular). The
**real live hazard** is
[`with_monotonicity(&b.comparisons, &monotone_pairs, 200)`](src/export.rs) at `export.rs:187`
(+ [`src/bt.rs`](src/bt.rs) `with_monotonicity`): it injects **200 synthetic "higher-q beats
lower-q, same codec" dummies per pair**, which pins q-ordering and will **inflate agreement with
any q-monotone metric like ssim2.** For the validation export: make the dummy count
**configurable** and report θ-vs-ssim2 **both with and without** the dummies (the honest number
is the no-dummy one; the dummied number bounds the inflation). Also ensure the sampler surfaces a
**range of pair difficulties** so the within-ref scale is well-connected. *(Non-photo note: the
cross-codec bytes-ratio>4× rule may over-trigger on line-art / flat content, where codec size
ratios are extreme at near-equal perceptual quality — but that branch isn't exercised today,
since only adjacent same-codec pairs are sampled.)*

- [ ] **4. Single-observer "solo expert" mode.** Bypass the crowd machinery, which assumes untrusted
raters: honeypot **fail-two-ends-session** (methodology §3.9; `p_honeypot = 0.083` at
[`src/sampling.rs:80`](src/sampling.rs)), session grading A–F →
`session_weight ∈ {0,0.5,1.0,1.5}` ([`src/grading.rs`](src/grading.rs)), and the 8-of-6
qualifier gate. For one trusted observer: **fix `session_weight = 1.0`**, **do not end the
session on honeypot fail** (keep honeypots as pure reliability telemetry), and **skip qualifier
gating**. Confirm the **desktop** path works end-to-end (the phone-first calibration / credit-card
slider, `web/src/calibration-onboarding.ts`, should run on desktop — verify). Lever #15
(expert routing) is ⚫ out-of-scope-by-design for the crowd study, so this is a small **new
mode**, not a reuse of an existing one.

- [ ] **5. Non-photo anchors.** Anchor reservation (lever #5; `p_anchor = 0.30` at
[`src/sampling.rs`](src/sampling.rs); `corpus_anchors` table) assumes **CID22 source overlap**
(README open decision + methodology §3.6). Non-photo has **none**. **Recommended:**
within-reference-only scales — each reference's θ is self-anchored at θ_reference = 0 via
`beta_to_quality`, and per-ref SROCC needs no cross-ref anchor. Set **`p_anchor = 0`** for this
run (or register per-reference pristine anchors only). No new anchor scheme is required as long
as the analysis stays within-reference.

- [ ] **6. v0.2 end-to-end smoke.** README/STUDY: v0.2 is 🟡 in progress, pending a real-device
end-to-end smoke (`cargo test --test smoke` + a human walkthrough). Land this before any real
observer run.

- [ ] **7. Sampling sanity on non-photo (corrected).** There is **no photo-tuned metric in the loop
today** — ASAP EIG ([`src/asap.rs`](src/asap.rs)) is driven by the **BT posterior fit from human
responses**, not a metric prior — so the "photo-tuned metric prior" concern is really about the
future PS-PC (#13). What *does* need checking on non-photo: verify the ingested conditions **span
the full quality range** (so the reconstructed scale connects), and confirm the **ASAP cold-start**
(`ASAP_MIN_OBS = 8`, [`src/sampling.rs:371`](src/sampling.rs)) behaves with the small
~8–12-conditions-per-reference item counts.

---

## Honest limits

**This is one observer, ~5 hours, a first-pass *sanity* validation — not a population MOS.** A single
observer's reconstructed scale carries that observer's idiosyncrasies; the result answers *"does ssim2
track this one careful human on non-photo content"*, **not** *"does ssim2 track the population."* A
proper population MOS needs ~15–30 observers per stimulus per **ITU-R BT.500**, which is a separate,
much larger study.

Within that limit it is **decisive either way**:

- **Pooled ~0.85 + no inverted or collapsed category** → ssim2 is a defensible non-photo oracle for
now; the `nonphoto` gate is measuring roughly the right thing, and we can say so with evidence
instead of by directive.
- **A category at ~0.70, or *inverted*** → the `nonphoto` gate (and every model tuned to rank well on
it) is optimizing toward the wrong target on that category. That is a first-order finding — it
justifies escalating that specific category to a multi-observer BT.500-grade study and, in the
meantime, down-weighting or fixing the gate there.

Either outcome retires an assumption we have been shipping on faith.