forrtproject / forrtproject/flora-extractor

Measured keyword rule fixes: phrase morphology, title-phrase rescue, TECHNICAL_*/GWAS exclusions

Open
#147 1 comment 0 reactions 0 assignees View on GitHub
stage-2
Dominant language
Python
Stars
2
Forks
1
Avg merge
1d 6h
Merged PRs (30d)
4

Description

Measured on real data during the #144 evidence pass (2026-08-03/04). All numbers are two-sided:
gold positives recovered **and** false-positive influx measured on real snapshot rows. Evidence
and machine-readable candidate tables are on branch `analysis/stage-b-eval`
(`analysis/stage_b_eval/vocab_holes_report.md`, `exclusion_narrowing_report.md`, plus the
`*_candidates.csv`). Under #146 these become policy-table entries rather than edits to a single
`keyword_verdict()` body; the measurements carry over either way.

Baseline: 1,564 admissions per million rows scanned.

### 1. Phrase-morphology bundle ("T2")

The keyword list misses 319 gold positives that contain no matching phrase. The hole is
morphological, not vocabulary: **274 of the 319 contain `replicat*`** in shapes the list does not
cover — "we successfully replicated", "sought/tried/wanted to replicate", etc. Categories:
`we replicate(d)` 97, matrix verb + `to replicate` 63, other stem sentence 39, GWAS-guard
kills 30, no textual signal 22, passive 17, econ Comment/Revisited 15, gerund 11, misc 25.

| bundle | gold recovered | extra admissions /M | goldneg hits |
| --- | ---: | ---: | ---: |
| **T2 (proposed)** | **186 of 319 (58%)** | **+9.3 (+0.6%)** | 0 |
| T4 (passive/gerund/3rd person) | 251 (79%) | +119.6 (+7.6%) | 0 |
| blanket abstract-stem → ambiguous | 278 | +10,819 (+692%) | 74 |

T2's two load-bearing pieces are a two-word gap between `we` and `replicate(d)`, and the
sought/tried/wanted family before `to replicate`. Do **not** take the passive, gerund, `replicab*`
or bare third-person arms — their noise is the biological-replicate and future-work senses, which
have no compact shape.

Ceiling for phrase work is 293: 26 of the 319 (22 signal-free, 4 `.supp` stubs) carry nothing a
keyword rule can use.

### 2. Title-phrase rescue

Where an exclusion pattern fired **only in the abstract** while the title carries a replication
phrase, demote to `ambiguous` instead of `negative`:
**15 gold positives for ~102 extra rows across the full 510M-row snapshot**, 0 gold-negative hits.
Needs `is_non_scholarly_context()` to also return the match position.

### 3. TECHNICAL_OBJECT / TECHNICAL_VERB narrowing

These fire on "replicate the code/data/model" — the literal description of a computational
reproduction. **13–18 gold positives for ~1.5–2k extra rows corpus-wide.** Independently confirmed
from the reproduction side: they kill 3 of the 5 indexed reproductions the pipeline currently
loses (`reproduction_coverage_report.md`).

### 4. GWAS: scope ruling needed before the exemptions ship

`_GWAS_GUARD` (a phrase guard) and `BIOLOGICAL_OF` (a row killer) together suppress 59 genetics
gold positives. Classified against the internal-replication rule in `shared/prompts.py`:
**47 qualify (80%), 6 are two-stage internal designs, 6 unclear, 0 molecular.**

Maintainer has ruled that internal (two-stage discovery + own replication cohort) designs are
**out** — consistent with the written rule. Genuine independent replications of previously
published genetic associations are **in**. Proposed:

- **BO1** — stop `genome(s)`/`genomic` matching inside `genome-wide`: recovers 20 (17 qualifying,
1 internal, 2 unclear), zero virology papers, ~204 projected extra rows corpus-wide.
- **G1 minus the `replicated\s+the\s+association` alternative** — recovers 12, **all 12
qualifying, 0 internal**. (G1 as originally written recovers 19 but leaks 2 internal designs
through that one alternative.)

Note the guard still misses 11–16 genuine qualifying papers whichever variant ships, because they
put the attribution in a different sentence from the "we replicated…" claim. No same-sentence
regex can reach those; they are a screen judgment, not a keyword one.

### Leave alone (measured)

- **Stage A stem gate** — misses only 41 of 7,505 gold positives (0.55%); every candidate new stem
measured at 0.00–0.10 gold per 1k extra admissions. A new stem in the Stage A mask alone admits
literally nothing (measured 0) unless the title-stem arm widens too.
- **EDITORIAL_ARTIFACT** (all 15 "kills" are PeerJ review/decision DOIs), **DATA_AVAILABILITY**
(7 of 8 are data deposits), **STRUCTURAL** (0.1 gold per 1k extra rows), **BIOLOGICAL** (largest
protective element; 0/20 sampled suppressions in scope).
- **Reproduction vocabulary** — already covered; all 22 measured candidates recover ≈nothing that
is not already caught.

### Acceptance

Each narrowed pattern ships with a test carrying the paper it was blocking **and** a near-miss it
must still block. Re-run the gold measurement and report the new counts, stratified as in #144's
gold-corpus caveat (the 62% `allrep_llm` share means these recall gains are upper bounds).

Related: #44 (second pass over hard-rejected buckets), #143, #146.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the measured reports and candidate tables on branch analysis/stage-b-eval, then inspect the policy-table work from #146, is_non_scholarly_context(), and the GWAS rules in shared/prompts.py. Add a test for each narrowed pattern with its blocking paper and near-miss, then rerun the gold measurement with the #144 stratification; done means the agreed recall and false-positive counts are reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.