forrtproject / forrtproject/flora-extractor

[Stage 3] Parse-method scoring needs redesign (ref-count domination + inert tie-breaker)

Open
#43 2 comments 0 reactions 0 assignees View on GitHub
enhancement stage-3
Dominant language
Python
Stars
2
Forks
1
Avg merge
1d 6h
Merged PRs (30d)
4

Description

## What / Where

`shared/pdf_parsing.py` — `score_parse_result()` (lines 379–399) picks the parse method whose text is fed to the LLM:

```
score = refs × 300 + abs_len + int_len × 2 + min(raw_len // 5, 1000)
```

Two problems make this scoring unreliable:

1. **`refs × 300` dominates but reference counts are not comparable across parsers.** MarkItDown's reference count comes from a loose regex, `_md_references()` (lines 288–309), which splits on numbered/bulleted patterns and caps at 60 entries. A garbled MarkItDown parse can manufacture ~60 pseudo-references and score `60 × 300 = 18000`, beating a clean GROBID parse with fewer but genuine structured refs. Reference *count* is treated as a proxy for overall extraction quality across methods that count references in fundamentally different ways.

2. **The `min(raw_len // 5, 1000)` tie-breaker is a constant.** Every parser caps `raw_text` at 5000 chars (`raw_text[:5000]` at lines 99, 142, 220, 373), so `raw_len // 5` is 1000 for any non-trivial parse. The term contributes a flat 1000 to every candidate and breaks no ties.

## Why it matters

The winning parse's `abstract + intro` is what the outcome and DOI-resolution LLMs actually read. A scoring function that can be gamed by a noisy parser's inflated ref count — and whose tie-breaker is inert — means the LLM can be fed worse text than an available cleaner parse. This silently degrades Stage 3 extraction quality with no visible signal.

## Proposed fix sketch

- Score against the **consumer's actual need**: for outcome extraction, weight abstract/intro *quality* (coherence, length, section detection), not ref count; use references only for the citation-matching path that needs them.
- **Normalize reference counts per parser** (or gate the ref bonus on structured-metadata methods like GROBID/openalex_xml, not the MarkItDown regex).
- Remove or redesign the inert raw-text tie-breaker (e.g. raise the char cap, or use a real signal).
- **Validate the weights empirically** against a small gold set of PDFs where the best parse is known.

Future work, not urgent — flagged by the maintainer for redesign.

Found in the 2026-07-16 end-to-end pipeline review.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in shared/pdf_parsing.py at score_parse_result() and inspect _md_references() plus the parser raw_text caps cited in the issue. Define a consumer-focused, parser-normalized scoring design, remove the inert tie-breaker, and validate the chosen weights against a small gold set where the best parse is known.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.