Stage B and beyond: staged processing of the snapshot corpus (supersedes screening-ladder rework plan)

# Stage B and beyond: staged processing of the snapshot corpus, evidence-based

## Purpose

This issue supersedes the previous version of #144. It leads with the proposed design; the evidence behind every number follows in Part II. All figures were measured on this checkout (commit `3c12ba3`) against real data: 10.9M snapshot rows for population structure, 5.6M for exclusion-pattern behaviour, 2.9M for vocabulary candidates, the 7,505-paper FLoRA gold corpus, 1,333 gold negatives, and the 93-report reproduction entry sheet. The draft was independently reviewed by a second model against the codebase; its corrections are folded in.

---

# Part I — The proposal

## Objective

Maximise new validated FLoRA entries per dollar **and per human validation hour**, with recall tracked against the gold corpus. The binding constraint is not LLM spend but validator time: every false positive that survives the screen costs a human minutes in the queue. Precision at admission is therefore worth more than any per-token saving, and no arm is processed at volume before its true-positive density is measured — the known-gold hit rates below (5.1% vs 0.09%) justify only the *order* of exploration, not full-arm commitments, because FLoRA's discovery history plausibly over-represents explicit replication language.

Each wave should carry explicit stop/go criteria — e.g. validated entries per $100 and validator minutes per accepted entry — evaluated on the tranche before the next is funded.

## Design principle: waves, not one pass

The survivor pool persists every Stage A survivor, so Stage B changes re-run locally from the pool — no rescan. We exploit that: process the corpus in **waves ordered by measured yield**, and between waves use audited negatives to shrink the pool. Each wave's spend is decided with the previous wave's yield in hand.

## Wave 0 — prerequisites and deterministic fixes

**0a. Persist admission provenance (schema change, blocks everything else).** `candidates.csv` currently records only `source=openalex_snapshot`; `_row_from_snapshot()` drops the concept hit, OpenAlex `type`, the keyword outcome, and the reproduction-vocabulary flag. Without them, arms cannot be selected downstream, Stage 2 cannot treat concept-admitted rows differently, and datasets cannot be filtered. Add these four fields (or a sidecar keyed by row identity) to the Stage 1 contract: `shared/schema.py`, indexes, merge behaviour, tests. This is the one change every wave depends on.

**0b. Rule changes**, each priced two-sided (gold positives recovered vs. false-positive influx measured on real snapshot rows; corpus-wide figures are projections from the sampled partitions and carry sampling error):

1. **Phrase-morphology bundle (T2).** The 319 no-phrase gold misses are overwhelmingly morphology, not vocabulary: 274 contain `replicat*` in shapes the phrase list doesn't cover ("we successfully replicated", "sought to replicate"). T2 recovers 186 of 319 (58%) for **+9.3 admissions per M rows scanned** (+0.6% on the 1,564/M baseline; ~4.7k rows projected corpus-wide) and zero hits on the screened gold negatives. The next tier (passives/gerunds) reaches 79% at 13× the influx — not proposed.
2. **Title-phrase rescue.** Where an exclusion pattern fired only in the abstract and the title carries a replication phrase, demote to `ambiguous`: **15 gold positives for ~102 projected extra rows corpus-wide.**
3. **TECHNICAL_OBJECT / TECHNICAL_VERB narrowing.** 13–18 gold positives for ~1.5–2k projected extra rows. Independently confirmed from the reproduction side: these patterns fire on "replicate the code/data/model" — the literal definition of a computational reproduction — and killed 3 of the 5 indexed reproductions the pipeline loses today.
4. **Concept-flag propagation into Stage 2** (requires 0a). Stage 1 admits via concept regardless of keyword verdict; Stage 2 terminally kills those rows. Concept-admitted rows with a negative verdict become `needs_review`. Regression test: a concept-only negative becomes `needs_review` while an otherwise-identical non-concept negative stays `false_positive`.
5. **Dataset routing** (requires 0a). 15–33% of every arm is `type == dataset` (Zenodo/Dataverse deposits — one sampled row is titled `EU27.tab`). Route to a quarantine file rather than dropping outright — curated reproduction reports can carry repository DOIs, and OpenAlex `type` misclassifies — and audit a sample before any terminal policy.
6. **Leave alone, with evidence:** the Stage A stem gate (misses only 41/7,505 gold positives; every candidate new stem measured at 0.00–0.10 gold per 1k extra admissions); the EDITORIAL_ARTIFACT, DATA_AVAILABILITY, STRUCTURAL and BIOLOGICAL exclusions (their gold "kills" are almost all junk records — PeerJ review DOIs, data deposits — the record is wrong, not the pattern); and reproduction vocabulary (already covered, see Wave R).

Each narrowed pattern ships with the paper it was blocking and a near-miss it must still block.

**Open scope decision (maintainer):** ~40–50 gold positives are GWAS locus-replication papers that `_GWAS_GUARD` and BIOLOGICAL_OF deliberately suppress. The gold set says in-scope, the code says out; exempting `genome-wide` costs ~200–400 projected extra rows. Measurement cannot settle whether FLoRA wants GWAS replications — this needs a ruling before Wave 0 ships.

Net effect of Wave 0 (excluding GWAS): keyword recall on the gold corpus rises from ~94% to ~97%, at well under +1% admissions.

## Wave R — the reproduction door (parallel, independent)

The reproduction entry sheet (93 distinct reports) shows vocabulary is a non-problem: 90% are OpenAlex-indexed and Stage 1 already admits 99% of those. What's missing is structural:

- **A curated-source harvester** for OSF / I4R / HAL / EconStor reports → `candidates.csv` with new source tags added to `CURATED_SOURCES` (the set is exact source strings, currently `i4r`, `bob_reed`, `backfill_old_pipeline` — new tags must be configured, and the `non_article_doi()` interaction tested, since it runs before the curated bypass). ~10% of the reproduction corpus is not in OpenAlex at all and is unreachable by any snapshot work. Note the bypass covers Stage 2 and the cheap pre-screen only: curated rows still face the validated Stage 3 screen, which is the desired behaviour.
- Wave 0 items 3–4 fix the indexed losses.

## Pilot before commitment

Three 1,000-row/arm random samples are already drawn from 40 partitions (reservoir sampling, seed 20260803; files to be committed with the eval artifacts). The pilot, not the known-gold rates, establishes per-arm true yield:

- Run the Tier-2 screen over all 3,000 rows (~$6–12).
- Human-label a **random sample of discards as well as proceeders** (labelling only proceeders would measure precision, not density, and would assume screen recall is perfect). Predefine the estimand, sample sizes per arm, and adjudication for unclear cases; stratify arm A on the reproduction-vocabulary flag.
- During the same pass, measure overlap of admissions with the existing 2.58M `candidates.csv` per identity key — gross admissions are not unique new rows, and all cost projections should be restated on unique Stage-3-eligible rows.

Output: per-arm density and screen proceed/recall estimates with intervals, dedup-adjusted volumes, and a tranche budget worksheet (screen price, proceed rate, downstream calls, OpenAlex cost, validator minutes).

## Wave 1 — the phrase arm, in tranches

Arm A (`keyword_verdict == positive`), `type` in {article, preprint}, stratified on the reproduction-vocabulary flag (half of arm A carries it; it is a regex heuristic, not a validated record type — the strata must be reported separately, not averaged). Projected ~375k gross before dedup and type filtering; the first tranche (~10–25k, strictest phrases first) runs only after the pilot confirms its yield, and each subsequent tranche is gated on the previous one's stop/go numbers. Expected screen bill at measured rates: **$2–4 per 1,000 rows**; ladder + outcome adds ~$11/1,000 *of proceeders* at measured output-token rates.

## Between waves — mine the negatives, carefully

Wave 1's `not_a_replication` rows are **model negatives, not confirmed negatives** — decisions of the same gate whose errors the design is trying to control. Deriving deterministic shrinkers (venues, fields, title shapes) from them can compound the screen's blind spots (poorly-described replications, non-English records, unfamiliar disciplines). Therefore: derive shrinkers from human-audited samples, run every shrinker in shadow mode with a random audit of what it would exclude, keep a locked holdout, and give each the same two-sided pricing as Wave 0 before it becomes terminal.

## Wave 2+ — expand by measured yield

- **Concept-only arm** (~220k gross projected) after the Stage 2 concept fix; known-gold hit rate 0.68%, 7% of all gold hits.
- **Ambiguous arm last, and only top-sliced.** ~765k gross projected at 0.09% known-gold hit rate — a 55× gap to the phrase arm on known positives. Blanket-screening it (~$1.5–3k) is not justified unless the pilot finds materially higher true density.

## The cheap pre-screen (Tier 1)

The staged design dissolves the urgency: tranche-sized screen bills are tens to low hundreds of dollars, so the tier's saving no longer justifies pre-commitment risk on unevaluated strata. Position:

- **#139 must not merge with its current defaults** (`PRESCREEN_ENABLED=1`, `PRESCREEN_MODE="discard"` — shadow exists but is not the default). Either default to `shadow` or bypass `keyword_verdict == ambiguous` rows before it lands. Flipping the default does not reopen prior `prescreen_discard` rows — a migration step (`--rescreen` over the set-aside) is part of the change.
- Revisit the tier only if a full ambiguous-arm pass is ever justified — and then only after a purpose-built, human-labelled title-stem gold set plus a shadow run. The existing evaluation sets contain 17 ambiguous rows total; the AND-gate discards 15 of them.

## Rebuilding the existing corpus

The ~2.5M rows already written `false_positive` under the old rule are not reclassified by resume logic, and `--admit-from-pool` is **append-only** (it skips rows matching any existing identity key and neither removes nor refreshes rows) — it is not a rebuild. The rebuild is a separate atomic procedure: build a fresh snapshot-candidate artifact from the pool → merge with preserved non-snapshot/curated sources under an explicit source-precedence policy → dedupe → validate counts and schema → atomically replace `candidates.csv` → rebuild its index → full Stage 2 re-filter into a fresh `filtered.csv` + index (deterministic, wall-clock only). For Stage 3, prior resolved rows and set-asides stay settled on resume; reopening screen-era decisions is explicitly `--rescreen`, and which set-asides are reopened must be stated in the rebuild PR. Schedule with Wave 1.

## Merge order

1. Fix #139's default (shadow, or the ambiguous bypass) — before or with anything else.
2. Land #142 (`feat/snapshot-scan`), then #143 (`feat/keyword-ladder`, a clean one-commit stack on #142; no textual conflicts with #139 — the interaction is purely semantic).
3. Wave 0a (provenance schema), then Wave 0b rule changes with their two-sided tests.
4. Harvester (Wave R) independently.
5. Pilot → tranche 1 of Wave 1 → stop/go from there.

Each wave graduates into its own implementation issue with exact commands, acceptance criteria, and row-count invariants; this issue fixes the design and the evidence.

---

# Part II — The evidence

## Population structure (40 partitions, 10,943,785 rows scanned = 2.1% of the 510M-record corpus)

| | admitted | % of admitted | projected gross, full corpus | known-gold hit rate | share of gold hits |
|---|---:|---:|---:|---:|---:|
| A positive | 8,027 | 27.5% | ~375k | **5.1%** | 89.7% |
| B ambiguous | 16,408 | 56.3% | ~765k | **0.09%** | 3.3% |
| C concept-only | 4,714 | 16.2% | ~220k | **0.68%** | 7.0% |
| total | 29,149 | 0.27% of scanned | **~1.36M** | | |

Projections scale the sampled rate to 510M records; partition variance is high (an independent 4-partition measure gave a survivor rate 40% lower), so treat these as central values with substantial uncertainty, and note they are **gross admissions before dedup** against the existing 2.58M-row corpus — overlap is unmeasured and is a pilot deliverable. The Stage A survivor pool projects to ~4–6M rows. The gate recovers ~79% of known FLoRA positives present in the scanned slice. Hit rates are floors and are biased toward FLoRA's historical discovery style (explicit replication language); they order exploration, nothing more. Arm descriptives: abstracts 89.5/68.2/71.8%; `type == dataset` 21.2/15.2/32.7%; reproduction-vocabulary flag 49.9/0.0/0.0%.

## Cost model, corrected

The widely-quoted $1.75/1,000 screen figure reproduces only as a **flex-tier** number. Rebuilt from measured tokens (~2,635 input/voter at 4.80 chars/token, validated against recorded API counts) and repo-recorded prices: **$3.72/1,000 standard, $1.86 with flex** (both voters). Downstream, the resolution ladder + outcome coding run ~**$11/1,000 of proceeders** — 3–6× the screen — dominated by output tokens (the heavy model emits ~3,500 tokens/call, ~20× the JSON answer; `GEMINI_THINKING_LEVEL` is unset, and no price for the heavy model is recorded in-repo, so this block is placeholder-priced). The single largest prompt is the never-truncated reference-list pick, not full text. OpenAlex API spend at the one recorded figure ($5–15 per 2,000 rows) is $2.5–7.5/1,000 — potentially comparable to all LLM spend. Cost levers, in order: tranche sizing by measured yield > thinking-level cap and reference-prompt bounds > flex > any screening tier.

If all ~1.36M gross admissions were processed in one pass, the envelope is roughly $5k–25k — the width driven by dedup overlap, per-arm proceed rates, and the unrecorded heavy-model price. The staged design exists so that this number is never spent blind.

## Vocabulary holes (319 no-phrase gold misses; candidates measured on 2.9M snapshot rows)

Stage A's stem gate misses only 41 of 7,505 gold positives (0.55%) — the hole is entirely in the shared phrase list, and it is morphological: 274/319 contain `replicat*`. Categories: "we ⟨adv⟩ replicate(d)" 97; matrix verb + "to replicate" 63; other stem sentence 39; GWAS-guard kills 30; no textual signal 22; passive 17; econ Comment/Revisited 15; gerund 11; misc 25. Anticipated categories that turned out empty: no-abstract (0), non-English (1). Pricing (gold recovered / incremental admissions per M scanned / gold-negative hits): T2 bundle **186 / +9.3 / 0**; T4 bundle 251 / +119.6 / 0; blanket abstract-stem→ambiguous 278 / +10,819 / 74 (rejected). Ceiling for phrase work: 293 (26 misses carry no usable signal).

## Exclusion patterns (121 gold kills; census of all 3,785 exclusion-firing survivors within 5.6M sampled rows)

Per-pattern protective value (rows suppressed per M scanned / sampled in-scope casualties): BIOLOGICAL 96.9 / 0/20 · BIOLOGICAL_OF 52.1 / 0/20 · STRUCTURAL 11.6 / 0/20 · DATA_AVAILABILITY 8.0 / 0/20 · TECHNICAL_OBJECT 5.5 / ~1/20 · TECHNICAL_VERB 3.2 / 3/18 · EDITORIAL_ARTIFACT 2.9 / 0/16. Only 2 of 300 BIOLOGICAL_OF firings were the GWAS genre — and both were themselves GWAS replication papers. Best trades: title-phrase rescue (15 gold / ~102 projected rows / 0 goldneg); `genome-wide` exemption (20–21 gold / ~204 projected rows / scope question); TECHNICAL_* tightening (13–18 gold / ~1.5–2k projected rows). EDITORIAL_ARTIFACT's 15 "kills" are all PeerJ review/decision DOIs; DATA_AVAILABILITY's are 7/8 data deposits — leave both alone.

## Reproductions (93 distinct reports from the entry sheet)

84/93 (90%) OpenAlex-indexed (DOI presence predicts indexing almost perfectly: 97.7% vs 0%). Of the indexed: Stage A gate 98.8%, Stage 1 admission 98.8%, end-to-end survival to Stage 3 94%. The five losses: 3 TECHNICAL_* misfires (fixed by Wave 0 items 3–4), 1 deposit DOI in the sheet (correct behaviour, bad data), 1 genuine vocabulary-free comment note. All 22 measured reproduction-vocabulary candidates recover ≈nothing not already caught; I4R boilerplate never appears in OpenAlex titles/abstracts. The unreachable 9 reports (OSF/HAL/EconStor) all carry clear vocabulary — the harvester is the fix, not keywords.

## Corrections to the previous version of this issue

Recorded so they are not re-invented: the "322" no-phrase misses reproduce as 319 (121 exclusion kills match exactly). §7's volume/cost numbers were measured on the old `candidates.csv`, which contains zero snapshot rows; on real Stage A output the ambiguous:positive ratio is ~2.4:1, not ~1.1:1, and "~$310 per corpus pass" does not transfer. The $1.75/1,000 screen figure is unsourced in-repo and holds only under flex pricing. #139 defaults to `PRESCREEN_MODE="discard"`, not shadow. The prior Tier-1 economics ($30 net on 49.8k rows) do not describe the snapshot population in either direction — the staged design replaces that debate with per-tranche decisions.

## Provenance

Measurement reports, candidate tables and pilot samples (vocab_holes_report.md, exclusion_narrowing_report.md, reproduction_coverage_report.md, pilot_sampling_notes.md + per-arm CSVs) were produced in-session on 2026-08-03 and will be committed under `analysis/stage_b_eval/` with the Wave 0 PR. Partitions sampled by deterministic stride from the 2,446-file manifest; all gate/verdict computations used production code imported from this checkout. A second model reviewed the draft against the codebase; its findings (arm-projection arithmetic, the missing provenance columns, append-only `--admit-from-pool`, pilot estimand, curated-bypass scope) are incorporated above.

Start with shared/schema.py and _row_from_snapshot(), then read the interactions with #139, #142, and #143 described in the proposal. This issue is a design and evidence record rather than a self-contained implementation task; it is done when each wave has its own implementation issue with exact commands, acceptance criteria, and row-count invariants.

Written by the indexing model from the issue text.