forrtproject / forrtproject/flora-extractor
Develop the reproduction routing rule on its own evidence
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
## Why this is its own issue
The Stage-2 rule book v2 proposal (`redesign/rulebook_v2.html`) rebuilds Stage 2 as a
whitelist and deliberately restricts it to **replication** vocabulary. Reproduction is a
different genre with a different precision profile, and folding it into the replication rules
means the two get promoted and measured together when their evidence has nothing to do with
each other.
Nothing measured for v2 says anything useful about reproduction. In the 2,895-positive FLoRA
corpus used to size the replication arms, `comput* reproduc*` fires on **4** rows and the
anchored `reproduction of the … results` pattern on **3**. There is no basis there for a
routing decision.
## Current state (unchanged by v2)
`filter/spec/phrase-reproduction.json` — ten anchored patterns, live, `screen_cheap`,
precedence 262, `vocabulary: reproduction`. v2 carries it over **verbatim and live** so the
reproduction stream does not go dark while this issue is worked. Do not fold it into the new
replication rules.
## What this issue has to settle
**Scope decisions first — these are definition questions, not measurements:**
- [ ] Is a computational reproduction (same data, same analysis, re-run) a FLoRA record on the
same footing as a replication? Stage 3 already codes it on two axes
(`outcome_computation` / `outcome_robustness`, see CLAUDE.md), so the answer is
probably yes — but the routing rule should be built against a stated definition.
- [ ] Is CS reimplementation / benchmarking in scope? *"We reimplemented and could not
reproduce the reported accuracy."*
- [ ] Are robustness/specification-curve papers reproductions, or a third thing?
- [ ] Are CODECHECK certificates and artifact-evaluation reports records, or artifacts
pointing at records? (Relevant to the `no-codable-text` discard in v2's A3a: a
certificate may have no codable text.)
**Then the vocabulary, with evidence, arm by arm:**
- [ ] `reproduction of` (bare) — known bad: 8% precision over 15,440 rows, 0 of 15 sampled
hits in our sense (#137). Animal breeding, social reproduction, epidemiological R₀.
- [ ] `re-analysis of` / `re-analys*` / `reanalyz*` — dominated by secondary analyses asking
*new* questions of old data. Needs a precision sample before any pile.
- [ ] `comput* reproduc*` (generic) — workflow engineering, containers, pedagogy, policy.
- [ ] `re-estimat*`, `reimplement*`, `rerun`, "using the authors' data and code we …" —
candidate probe arms, no counts yet.
- [ ] "robustness reproducibility/replicability" — methodological or metric-oriented as often
as not.
## Evidence this needs that v2's does not have
The replication design could lean on 2,895 known positives with abstracts. There is no
equivalent reproduction corpus in the repo. So this issue needs to **build its labelled set
first**:
- reproduction records already in FLoRA (`data/flora.csv` / the entry sheet) — count them and
check whether the number supports anything;
- CODECHECK's public register and journal reproducibility-check collections (e.g. the AEA data
editor's reports, *Economic Journal* / *ReScience* style venues) as independent seed lists;
- a hand-labelled sample drawn from what today's `phrase-reproduction` routes.
Same rules as v2: human labels are the authority, LLM output is not a reference label, and
per-arm sampling with the denominator recorded (#151).
## Acceptance
A `phrase-reproduction` replacement is ready to promote when each arm has its own count on the
survivor pool, its own human-labelled precision estimate with a stated denominator, and a
scope definition that says which of the four genres above it is trying to catch.
## Related
- `redesign/rulebook_v2.html` §3.7 (the carve-out) and §1 (method and biases)
- #137 (bare "reproduction of" precision), #146 (the filter engine), #151 (gold corpus
circularity)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with filter/spec/phrase-reproduction.json and redesign/rulebook_v2.html §3.7, then inspect data/flora.csv, CLAUDE.md, and the related issues for existing labels and routing behavior. Build a labelled sample from the listed seed sources and current routes, recording denominators per arm. Done means each arm has a human-labelled precision estimate and a stated scope definition supporting the replacement rule.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100