forrtproject / forrtproject/flora-extractor

[Stage 2] Second pass over hard-rejected buckets for high recall

Open
#44 1 comment 0 reactions 0 assignees View on GitHub
enhancement stage-2
Dominant language
Python
Stars
2
Forks
1
Avg merge
1d 6h
Merged PRs (30d)
4

Description

## What / Where

Stage 2 (`filter/`) hard-rejects large buckets at high confidence with **no LLM review and no re-examination path**:

- Rows with no replication phrase (`filter_evidence = 'no replication phrase detected'`, ~2.17M rows) are rejected by the rule filter and never seen by the LLM.
- Rows killed by an exclusion pattern are likewise rejected without LLM review.
- The only re-examination tool, `filter/refilter_fp.py`, revisits rows only for *newly added* replication phrases — it does not re-open the never-reviewed rejected buckets.

By contrast, rows that had a replication phrase but no citation *did* reach the LLM.

## Why it matters

Data point: of the ~132k "phrase without citation" rows that reached the LLM, **21.5% were readmitted**. That readmission rate on a bucket the rules were inclined to drop suggests material recall loss in the far larger buckets that are *never* reviewed at all. A high-recall pipeline cannot rely on rule rejections it never audits.

A specific known failure mode: exclusion patterns misfire on in-scope **computational reproductions**, e.g. "replicated the analysis code of Smith (2019)" — a valid reproduction that an exclusion pattern (biological/robustness language) can kill even though a replication phrase *and* an author-year citation are both present.

## Proposed fix sketch

- **Second-pass over the rejected bucket**: an embedding-similarity or cheap-LLM pass over a sample (or all) of the no-phrase / exclusion-rejected rows to estimate and recover false rejections.
- **Targeted readmission rule**: re-open rows where an exclusion pattern fired *but* a replication phrase **and** an author-year citation were also present — these are the exclusion misfires on in-scope reproductions.
- Quantify the false-rejection rate from a hand-labeled sample (see the empirical-validation tracking issue) so the recall cost is a measured number, not an assumption.

Found in the 2026-07-16 end-to-end pipeline review.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the rejection paths under filter/ and the existing filter/refilter_fp.py tool to understand which buckets are never reopened. Review the empirical-validation tracking issue mentioned in the proposal before choosing how to sample and label false rejections. Done means measuring the false-rejection rate and implementing the agreed second-pass and targeted readmission behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.