forrtproject / forrtproject/flora-extractor
Measured keyword rule fixes: phrase morphology, title-phrase rescue, TECHNICAL_*/GWAS exclusions
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
Measured on real data during the #144 evidence pass (2026-08-03/04). All numbers are two-sided:
gold positives recovered **and** false-positive influx measured on real snapshot rows. Evidence
and machine-readable candidate tables are on branch `analysis/stage-b-eval`
(`analysis/stage_b_eval/vocab_holes_report.md`, `exclusion_narrowing_report.md`, plus the
`*_candidates.csv`). Under #146 these become policy-table entries rather than edits to a single
`keyword_verdict()` body; the measurements carry over either way.
Baseline: 1,564 admissions per million rows scanned.
### 1. Phrase-morphology bundle ("T2")
The keyword list misses 319 gold positives that contain no matching phrase. The hole is
morphological, not vocabulary: **274 of the 319 contain `replicat*`** in shapes the list does not
cover — "we successfully replicated", "sought/tried/wanted to replicate", etc. Categories:
`we replicate(d)` 97, matrix verb + `to replicate` 63, other stem sentence 39, GWAS-guard
kills 30, no textual signal 22, passive 17, econ Comment/Revisited 15, gerund 11, misc 25.
| bundle | gold recovered | extra admissions /M | goldneg hits |
| --- | ---: | ---: | ---: |
| **T2 (proposed)** | **186 of 319 (58%)** | **+9.3 (+0.6%)** | 0 |
| T4 (passive/gerund/3rd person) | 251 (79%) | +119.6 (+7.6%) | 0 |
| blanket abstract-stem → ambiguous | 278 | +10,819 (+692%) | 74 |
T2's two load-bearing pieces are a two-word gap between `we` and `replicate(d)`, and the
sought/tried/wanted family before `to replicate`. Do **not** take the passive, gerund, `replicab*`
or bare third-person arms — their noise is the biological-replicate and future-work senses, which
have no compact shape.
Ceiling for phrase work is 293: 26 of the 319 (22 signal-free, 4 `.supp` stubs) carry nothing a
keyword rule can use.
### 2. Title-phrase rescue
Where an exclusion pattern fired **only in the abstract** while the title carries a replication
phrase, demote to `ambiguous` instead of `negative`:
**15 gold positives for ~102 extra rows across the full 510M-row snapshot**, 0 gold-negative hits.
Needs `is_non_scholarly_context()` to also return the match position.
### 3. TECHNICAL_OBJECT / TECHNICAL_VERB narrowing
These fire on "replicate the code/data/model" — the literal description of a computational
reproduction. **13–18 gold positives for ~1.5–2k extra rows corpus-wide.** Independently confirmed
from the reproduction side: they kill 3 of the 5 indexed reproductions the pipeline currently
loses (`reproduction_coverage_report.md`).
### 4. GWAS: scope ruling needed before the exemptions ship
`_GWAS_GUARD` (a phrase guard) and `BIOLOGICAL_OF` (a row killer) together suppress 59 genetics
gold positives. Classified against the internal-replication rule in `shared/prompts.py`:
**47 qualify (80%), 6 are two-stage internal designs, 6 unclear, 0 molecular.**
Maintainer has ruled that internal (two-stage discovery + own replication cohort) designs are
**out** — consistent with the written rule. Genuine independent replications of previously
published genetic associations are **in**. Proposed:
- **BO1** — stop `genome(s)`/`genomic` matching inside `genome-wide`: recovers 20 (17 qualifying,
1 internal, 2 unclear), zero virology papers, ~204 projected extra rows corpus-wide.
- **G1 minus the `replicated\s+the\s+association` alternative** — recovers 12, **all 12
qualifying, 0 internal**. (G1 as originally written recovers 19 but leaks 2 internal designs
through that one alternative.)
Note the guard still misses 11–16 genuine qualifying papers whichever variant ships, because they
put the attribution in a different sentence from the "we replicated…" claim. No same-sentence
regex can reach those; they are a screen judgment, not a keyword one.
### Leave alone (measured)
- **Stage A stem gate** — misses only 41 of 7,505 gold positives (0.55%); every candidate new stem
measured at 0.00–0.10 gold per 1k extra admissions. A new stem in the Stage A mask alone admits
literally nothing (measured 0) unless the title-stem arm widens too.
- **EDITORIAL_ARTIFACT** (all 15 "kills" are PeerJ review/decision DOIs), **DATA_AVAILABILITY**
(7 of 8 are data deposits), **STRUCTURAL** (0.1 gold per 1k extra rows), **BIOLOGICAL** (largest
protective element; 0/20 sampled suppressions in scope).
- **Reproduction vocabulary** — already covered; all 22 measured candidates recover ≈nothing that
is not already caught.
### Acceptance
Each narrowed pattern ships with a test carrying the paper it was blocking **and** a near-miss it
must still block. Re-run the gold measurement and report the new counts, stratified as in #144's
gold-corpus caveat (the 62% `allrep_llm` share means these recall gains are upper bounds).
Related: #44 (second pass over hard-rejected buckets), #143, #146.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the measured reports and candidate tables on branch analysis/stage-b-eval, then inspect the policy-table work from #146, is_non_scholarly_context(), and the GWAS rules in shared/prompts.py. Add a test for each narrowed pattern with its blocking paper and near-miss, then rerun the gold measurement with the #144 stratification; done means the agreed recall and false-positive counts are reported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100