PolicyEngine / PolicyEngine/microcosm-benchmarks
Promotion gates need an uncapped tail block (q90/q99 ratios, W1/sd) and fragility-vs-truth per imputed variable
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 0
- Forks
- 0
- Avg merge
- 29m
- Merged PRs (30d)
- 1
Description
Motivation (measured)
The imputation paper's harness (github.com/PolicyEngine/imputation-paper) initially certified a candidate whose imputed net-worth 99th percentile was 1.9–2.25× the holdout's: weighted energy distance, PRDC coverage, and a classifier two-sample test all tied it with a correct candidate, because capped pairwise geometry under-resolves heavy right tails. The fix was a fourth, uncapped tail block per imputed variable. Promotion gates built on target attainment plus geometry metrics will certify tail-broken bundles.
Proposal
Add to the incumbent-comparison promotion metrics, per imputed variable, computed on full weighted samples (no subsample cap) against the pinned reference:
- q90 and q99 ratios (candidate over reference; 1 is perfect; band calibrated by a sampling floor — the paper's floor wobbles to ~1.10 on a 920-record holdout),
- weighted Wasserstein-1 / reference sd,
- reweight fragility vs the reference's own fragility (closed form: worst-case single-record share of the aggregate under bounded multiplicative reweighting, κ=5; the target is the reference's exposure — far below it means missing tails, far above means landmines; measured range: OLS-style imputations understate true exposure by 7–11×).
Reference implementations: experiments/metrics.py (reweight_fragility, _weighted_quantile) and experiments/views.py (tail block) in the paper repo; the promotion rule in this repo's README ("beats the pinned incumbent on every promotion metric") then covers the tail by construction.
🤖 Generated with Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the promotion rule in README and inspect how incumbent-comparison metrics are assembled. Compare the target implementation with experiments/metrics.py and experiments/views.py in the referenced paper repository, especially reweight_fragility, _weighted_quantile, and the tail block. Done means the promotion metrics include the specified per-variable tail and fragility comparisons on full weighted samples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100