forrtproject / forrtproject/flora-extractor
[Stage 2] Second pass over hard-rejected buckets for high recall
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
## What / Where
Stage 2 (`filter/`) hard-rejects large buckets at high confidence with **no LLM review and no re-examination path**:
- Rows with no replication phrase (`filter_evidence = 'no replication phrase detected'`, ~2.17M rows) are rejected by the rule filter and never seen by the LLM.
- Rows killed by an exclusion pattern are likewise rejected without LLM review.
- The only re-examination tool, `filter/refilter_fp.py`, revisits rows only for *newly added* replication phrases — it does not re-open the never-reviewed rejected buckets.
By contrast, rows that had a replication phrase but no citation *did* reach the LLM.
## Why it matters
Data point: of the ~132k "phrase without citation" rows that reached the LLM, **21.5% were readmitted**. That readmission rate on a bucket the rules were inclined to drop suggests material recall loss in the far larger buckets that are *never* reviewed at all. A high-recall pipeline cannot rely on rule rejections it never audits.
A specific known failure mode: exclusion patterns misfire on in-scope **computational reproductions**, e.g. "replicated the analysis code of Smith (2019)" — a valid reproduction that an exclusion pattern (biological/robustness language) can kill even though a replication phrase *and* an author-year citation are both present.
## Proposed fix sketch
- **Second-pass over the rejected bucket**: an embedding-similarity or cheap-LLM pass over a sample (or all) of the no-phrase / exclusion-rejected rows to estimate and recover false rejections.
- **Targeted readmission rule**: re-open rows where an exclusion pattern fired *but* a replication phrase **and** an author-year citation were also present — these are the exclusion misfires on in-scope reproductions.
- Quantify the false-rejection rate from a hand-labeled sample (see the empirical-validation tracking issue) so the recall cost is a measured number, not an assumption.
Found in the 2026-07-16 end-to-end pipeline review.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the rejection paths under filter/ and the existing filter/refilter_fp.py tool to understand which buckets are never reopened. Review the empirical-validation tracking issue mentioned in the proposal before choosing how to sample and label false rejections. Done means measuring the false-rejection rate and implementing the agreed second-pass and targeted readmission behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100