Screening and annotation send the bibliography to the model: ~35% of input is never used

Full-text screening and annotation send the entire article to the model, including the bibliography and front matter. Neither is used by any screening or annotation criterion.

Measured

Across 298 pubget articles:

Per-call input, measured from the token accounting added in autonima/llm/usage.py:

Those two stages are 61% of total project cost. Removing ~35% of their input saves roughly $0.010 per full-text call and $0.011 per annotation call — on the order of 20% of a whole project, with no behavioural change.

For scale: the nine-project validation benchmark costs ~$194 to reproduce from scratch, so this is ~$40 of pure waste per full corpus run, and it recurs on every re-run.

Where

study.load_full_text() feeds PromptLibrary.get_fulltext_screening_prompt (and the annotation path when study_fulltext is in metadata_fields). The strip belongs at load time so every consumer benefits, not in each prompt builder.

For JATS the elements are unambiguous (<ref-list>, <front>). For the HTML/text sources it is messier and probably wants a heuristic on a trailing "References"/"Bibliography" heading — worth doing the JATS case first since it is exact and covers most of the corpus.

Worth checking while in there

A stronger version would send only the sections screening actually needs (Title, Abstract, Methods, Results). I attempted to measure that share and got an implausible 15%, because nested <sec> elements double-count — so the number needs a proper section-aware pass before anyone relies on it. The direction is clearly right; the magnitude is not yet established.

Related

Complements #68: once the bibliography is gone, the invariant criteria block becomes a much larger share of the remaining prompt, so prefix caching pays off more.

Start with study.load_full_text() and trace how it feeds PromptLibrary.get_fulltext_screening_prompt and the annotation path when study_fulltext is in metadata_fields. Review autonima/llm/usage.py for the token accounting, then implement the exact JATS removal of and at load time. Done means those elements are excluded without behavioral changes, with the measured input reduction verified.