[FT] IRT-based scoring as alternative ranking method for fairer model comparison

## Issue encountered

Simple averaging across benchmarks produces **misleading rankings** when:

- Benchmark difficulty varies (up to **12×** in discrimination)
- Evaluation matrices are sparse (e.g. only 31% of cells filled)
- Models selectively report on easier benchmarks (reporting bias **r = −0.64**)

**Scale of the problem:**

| Dataset | Models | Result |
|---|---|---|
| VLA robotics (Evo-SOTA.io) | 133 models, 18 benchmarks | Up to **78-rank shift** after IRT correction |
| Gemini 3.5 Flash (Google I/O) | 6 models, 14 benchmarks | Simple avg **ρ = 0.24** vs IRT **ρ = 0.81** (100-fold CV) |
| Cross-domain replication | 7 independent domains | IRT maintains **ρ ≥ 0.993** |

Math-Verify addressed fairness in answer validation — this addresses a different bias source: **benchmark difficulty weighting and missing data**.

> NIST CAISI adopted the same IRT-based approach in their May 2026 DeepSeek V4 Pro evaluation.
> [Source: nist.gov/caisi](https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro)

## Solution/Feature

An optional **IRT (Item Response Theory) scoring mode** that jointly estimates:

- **Model ability (θ)** — corrected for which benchmarks were attempted
- **Item difficulty (b)** — learned from response patterns across all models
- **Item discrimination (a)** — how well each benchmark separates strong from weak models

Performance: runs in JavaScript, **< 100ms**, CC BY 4.0 licensed, zero server infrastructure needed.

- 📄 Paper: [arXiv:2605.11205](https://arxiv.org/abs/2605.11205)
- 🔗 Live demo: [psycrank.com](https://psycrank.com)
- 💻 Source: [github.com/testofschool](https://github.com/testofschool)

## Possible alternatives

| Approach | Pros | Cons |
|---|---|---|
| Weighted averaging with manual weights | Simple to implement | Requires human judgment per benchmark, not principled |
| Bootstrap confidence intervals only | Addresses variance | Doesn't correct systematic difficulty bias |
| IRT as post-hoc analysis (not integrated) | Simpler integration | Lower adoption, users must run separately |