forrtproject / forrtproject/flora-extractor
[Stage 3] Parse-method scoring needs redesign (ref-count domination + inert tie-breaker)
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
## What / Where
`shared/pdf_parsing.py` — `score_parse_result()` (lines 379–399) picks the parse method whose text is fed to the LLM:
```
score = refs × 300 + abs_len + int_len × 2 + min(raw_len // 5, 1000)
```
Two problems make this scoring unreliable:
1. **`refs × 300` dominates but reference counts are not comparable across parsers.** MarkItDown's reference count comes from a loose regex, `_md_references()` (lines 288–309), which splits on numbered/bulleted patterns and caps at 60 entries. A garbled MarkItDown parse can manufacture ~60 pseudo-references and score `60 × 300 = 18000`, beating a clean GROBID parse with fewer but genuine structured refs. Reference *count* is treated as a proxy for overall extraction quality across methods that count references in fundamentally different ways.
2. **The `min(raw_len // 5, 1000)` tie-breaker is a constant.** Every parser caps `raw_text` at 5000 chars (`raw_text[:5000]` at lines 99, 142, 220, 373), so `raw_len // 5` is 1000 for any non-trivial parse. The term contributes a flat 1000 to every candidate and breaks no ties.
## Why it matters
The winning parse's `abstract + intro` is what the outcome and DOI-resolution LLMs actually read. A scoring function that can be gamed by a noisy parser's inflated ref count — and whose tie-breaker is inert — means the LLM can be fed worse text than an available cleaner parse. This silently degrades Stage 3 extraction quality with no visible signal.
## Proposed fix sketch
- Score against the **consumer's actual need**: for outcome extraction, weight abstract/intro *quality* (coherence, length, section detection), not ref count; use references only for the citation-matching path that needs them.
- **Normalize reference counts per parser** (or gate the ref bonus on structured-metadata methods like GROBID/openalex_xml, not the MarkItDown regex).
- Remove or redesign the inert raw-text tie-breaker (e.g. raise the char cap, or use a real signal).
- **Validate the weights empirically** against a small gold set of PDFs where the best parse is known.
Future work, not urgent — flagged by the maintainer for redesign.
Found in the 2026-07-16 end-to-end pipeline review.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in shared/pdf_parsing.py at score_parse_result() and inspect _md_references() plus the parser raw_text caps cited in the issue. Define a consumer-focused, parser-normalized scoring design, remove the inert tie-breaker, and validate the chosen weights against a small gold set where the best parse is known.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100