PolicyEngine / PolicyEngine/microcosm-benchmarks

Promotion gates need an uncapped tail block (q90/q99 ratios, W1/sd) and fragility-vs-truth per imputed variable

Open
#6 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

tier:standard
Dominant language
No language data
Stars
0
Forks
0
Avg merge
29m
Merged PRs (30d)
1

Description

Motivation (measured)

The imputation paper's harness (github.com/PolicyEngine/imputation-paper) initially certified a candidate whose imputed net-worth 99th percentile was 1.9–2.25× the holdout's: weighted energy distance, PRDC coverage, and a classifier two-sample test all tied it with a correct candidate, because capped pairwise geometry under-resolves heavy right tails. The fix was a fourth, uncapped tail block per imputed variable. Promotion gates built on target attainment plus geometry metrics will certify tail-broken bundles.

Proposal

Add to the incumbent-comparison promotion metrics, per imputed variable, computed on full weighted samples (no subsample cap) against the pinned reference:

  • q90 and q99 ratios (candidate over reference; 1 is perfect; band calibrated by a sampling floor — the paper's floor wobbles to ~1.10 on a 920-record holdout),
  • weighted Wasserstein-1 / reference sd,
  • reweight fragility vs the reference's own fragility (closed form: worst-case single-record share of the aggregate under bounded multiplicative reweighting, κ=5; the target is the reference's exposure — far below it means missing tails, far above means landmines; measured range: OLS-style imputations understate true exposure by 7–11×).

Reference implementations: experiments/metrics.py (reweight_fragility, _weighted_quantile) and experiments/views.py (tail block) in the paper repo; the promotion rule in this repo's README ("beats the pinned incumbent on every promotion metric") then covers the tail by construction.

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the promotion rule in README and inspect how incumbent-comparison metrics are assembled. Compare the target implementation with experiments/metrics.py and experiments/views.py in the referenced paper repository, especially reweight_fragility, _weighted_quantile, and the tail block. Done means the promotion metrics include the specified per-variable tail and fragility comparisons on full weighted samples.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.