forrtproject / forrtproject/flora-extractor
Pre-screen override: land the 16 widened signal patterns, re-measured against the screen_cheap pile
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
Salvaged from #139, which is closed as stale: the flag machinery it was built around
(`PRESCREEN_ENABLED`, `PRESCREEN_MODE`) was deliberately removed in #161, and the tier
moved out of `run_extract` into Stage 2's `filter/engine/tiers.py`. One part of that PR
never landed and is still worth having.
## What is missing from main
`_SIGNAL_PATTERNS` in `shared/prescreen.py` carries only its original block. The 16
patterns derived in #139 from the vocabulary of 7,505 FLoRA replications and
reproductions — plus the evidence for them — are on `origin/issue-130-prescreen-followup`
and nowhere else:
- the pattern block itself (~30 lines, applies cleanly; `shared/prescreen.py` auto-merges)
- `analysis/prescreen_eval/OVERRIDE_EVAL.md`
- `analysis/prescreen_eval/build_override_sets.py`, `eval_override.py`
- the case sets those scripts read
## Why it matters
`hard_signal()` is a bypass: it is what stops a 3B model getting the last word on a row
whose text states the design outright. A `prescreen_discard` is terminal, so recall here
is not commensurable with dollars. #139's measurement was that the override was catching
79.9% of positives, not the ~100% its derivation set suggested; the 16 patterns took that
to 94.7% (94.1% held out), for roughly $7 a corpus pass in screen calls the tier no
longer avoids.
Two boundaries were tested and declined, and should stay declined: widening
`replication of` to a list of study-like objects (+1.1 points for $6), and the bare
phrase `replication of` (+4 points for $44 and half the screen-confirmed negatives —
"replication of DNA" is the same eleven characters).
## This does NOT widen Stage 1
Worth stating because it is the obvious first question. Stage 1's gate is
`REPLICATION_STEM_PATTERN` in `filter/phrase_detection.py` — a bare stem alternation
(`replicat|replicab|reproduc|reanalys|…`) over the inverted index, unioned with two
OpenAlex concept ids. Every one of the 16 new patterns contains `replicat` or `reproduc`,
so every work they fire on is already in the survivor pool. The override changes which
rows the *cheap tier* may end, nothing about what is discovered.
## The work
The patterns are cheap to apply. The blocker is that the numbers above were measured on
the wrong population for the current design: 7,505 FLoRA papers with usable abstracts
plus 1,333 non-replications "that genuinely reach Stage 3" — from when the pre-screen sat
in front of everything entering Stage 3. Today the tier runs only over rows the rule book
routed to `screen_cheap`, a smaller population selected by rules that did not exist when
the eval ran. Both headline numbers move, and the trade is close enough (the $7 is about
a quarter of what the tier earns; #139 bracketed the tier's live saving at $1–28 a pass,
not the $30 previously quoted) that the wrong denominator could invert it.
So, in order:
1. Re-measure the override's bypass rate over the actual `screen_cheap` pile, and the
marginal screen cost of the rows it newly bypasses. No LLM calls needed for the
bypass rate — it is regex over abstracts.
2. If the trade holds, land the pattern block plus `OVERRIDE_EVAL.md` and the two
scripts, with the re-measured numbers replacing the Stage-3-era ones.
3. If it does not hold, land the subset that does, and record in `OVERRIDE_EVAL.md`
which patterns were dropped and why.
Branch `issue-130-prescreen-followup` is kept, not deleted — it is the only copy of the
patterns and the eval.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with shared/prescreen.py and the preserved issue-130-prescreen-followup branch, then read analysis/prescreen_eval/OVERRIDE_EVAL.md, build_override_sets.py, and eval_override.py. Re-measure the 16-pattern bypass rate and marginal screen cost on the current screen_cheap case sets. Done means landing the supported pattern subset and updated evaluation evidence, or documenting dropped patterns and the tradeoff.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100