PolicyEngine / PolicyEngine/snap-qc-sim

Required-n: what sample size makes tier assignment reliable

Open
#90 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
0
Avg merge
18m
Merged PRs (30d)
5

Description

The most decision-relevant number for the state-AG briefing Jonathan Meltzer is arranging (thread of 8/27), and the quantitative core of the 7 USC 2025(c)(1)(B) "valid statistical results" question. Also directly tests Bauer–Schanzenbach's asserted-but-not-analyzed claim that even CI-based approaches "will not solve" the problem. Strictly neutral: compute, don't recommend.

For each state: the QC sample size n* at which tier assignment reaches reliability targets (e.g., modal-tier probability ≥ 0.90/0.95 at the state's official rate), via the existing bootstrap machinery (resampling at m > n is already how the app prices extra audits — see simulate() extra_audits). Report:

  • n* per state vs current sample size (and the ratio) at both sampling-only and persistence-widened dispersion (the honest forward-looking case; widened n* may be infinite for boundary states — that's a finding, not a failure: process variance doesn't shrink with n).
  • The multi-year-averaging alternative: years of averaging k needed at current n (averaging cuts sampling variance ~1/k and transitory variance per the AR(1); the persistence layer's fitted parameters give the exact arithmetic).
  • The CI-lower-bound alternative (pre-2014 practice per Bauer–Schanzenbach): billing frequency under lower-bound rules at current n.

The federal-subsample layer (7 CFR 275.3 — the subsample minimum that doesn't scale) is NOT modeled in the repo; state that as a scope boundary, do not encode reg mechanics without reading the reg text (no-fabricated-mechanisms).

Artifact + memo + locks; sol review before merge (external stakes — these numbers go to lawyers).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing simulate() bootstrap machinery, including the extra_audits path, and inspect how persistence parameters and current sample sizes are represented. Produce the per-state reliability, averaging, and CI-lower-bound results, plus the requested artifact, memo, and locks; explicitly document the federal-subsample layer as out of scope and obtain sol review before merge.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.