blackjax-devs / blackjax-devs/tuningfork

Verdict-bearing full-distribution GT gate: per-dim W1/σ two-prong equivalence test

Open
#230 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3
Forks
0
Avg merge
7h 24m
Merged PRs (30d)
4

Description

Successor to the z-band work (#223). The mean-z point-estimate comparison is now advisory at ensemble scale (#226) because a point-null test hits the NHST trap — it rejects *any* fixed discrepancy (including the GT's own MC error) as resolution grows. This tracks the positive replacement: a verdict-bearing comparison of the full marginal distribution that is trap-free by construction.

**Design (ratified):** per-dim **Wasserstein-1 in GT-σ units** run as a two-sample **equivalence test** — two prongs:
- **max prong** — `max_d W1/σ` vs a *floor-of-max* null (catches concentrated bias; exact FWER)
- **frac prong** — fraction of dims over `max(floor_d, τ_sci)` with a bootstrap-calibrated, correlation-aware threshold (catches diffuse bias that max-only misses)

Floor is asymmetric in both effective sizes (`c_d·√(1/E_s+1/E_g)`, conservative `E = min(bulk,tail)-ESS`); **second-stage** (runs only after R̂/ESS/div pass — ESS estimators fail exactly when samplers struggle); per-dim k̂ guard routes heavy-tailed dims to trimmed-W1/PIT; whitened sliced-W1 as an advisory joint realm; SBC (per-parameter ranks) as the no-GT escape hatch. `τ_sci = 0.05σ` starting value (distributional re-derivation owed, non-blocking).

**Why it escapes the trap:** as resolution grows the floor → 0 but the fixed materiality bar stays — a clean sampler always passes, a biased one is caught *harder*. Decisive receipt: the honest-null the current gate FAILs at z=4.5 is correctly cleared (0.022σ, 0/144 dims over floor).

**Prerequisite (now met):** the max prong is resolution-bound on single-chain GT (floor-of-max 0.075–0.099σ > 0.05σ); it needs the multichain GT, which landed in #228.

**Open items:** τ_sci distributional re-derivation; floor-construction method pinned with a golden; joint-realm power (whitened sliced-W1 vs targeted Frobenius-Δcorr); W1-vs-KS/AD measured-power head-to-head.

Lands as a new module inside the post-#227 `_gate/` package.

— 🤖 Blackjax-devs AI TL

Contributor guide

Open the contributing guide

Research direction

Start by reading the post-#227 `_gate/` package and the multichain GT work in #228, then inspect the current gate behavior described in the issue. The new module should implement the ratified two-prong W1/σ equivalence comparison and reproduce the honest-null receipt that currently fails; resolve the listed open items and validate the golden floor construction before considering it done.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.