PubMed search silently truncates at 10,000 results; max_results above that is ignored

Summary

PubMed search silently truncates at ~10,000 results. Any max_results above that is ignored, and nothing warns that candidates were dropped.

Root cause

autonima/search/pubmed.py passes max_results straight through as retmax, with no paging:

handle = Entrez.esearch(
    db="pubmed",
    term=query,
    retmax=self.config.max_results,
    sort="relevance"
)

NCBI caps esearch retmax at 10,000, so a larger value is clipped by the API. esearch does return the true total in Count, but it is never compared against len(IdList), so the truncation is invisible.

Evidence

A project config with max_results: 20000 returned 9,999 studies on two separate runs:

9999  problem_solving/v1
9999  problem_solving/v2

Querying the same term and date window directly against E-utilities reports 12,075 matching records, so roughly 2,000 candidates were dropped without any indication.

Impact

Papers never retrieved cannot be recovered by any downstream stage, so this is a silent recall ceiling. For the run above, measured search recall against a gold standard was 0.881 where the untruncated query reaches 0.890 — so the practical cost happened to be small, but only because sort="relevance" kept most of the relevant records. That is luck rather than design: with a different query the dropped 2,000 could contain a much larger share of the target papers, and nothing in the logs or outputs would show it.

Most projects are unaffected (their queries return well under 10,000). It bites exactly the broad-query case where recall matters most.

Suggested fix

1. Page the request. Loop retstart in 10,000-record steps until len(IdList) >= Count or max_results is reached. This makes max_results mean what it says.
2. At minimum, warn. Read Count from the response and log a warning when Count > len(IdList), naming both numbers, so truncation is visible even if paging is not implemented.
3. Consider whether sort="relevance" is the right default when truncation is possible — it makes the retained subset non-obvious. sort="pub_date" at least makes it reproducible and explainable.

Option 2 alone would have surfaced this immediately.

Start in autonima/search/pubmed.py at the Entrez.esearch call and trace how Count and IdList are handled after the response. Verify the behavior when max_results exceeds 10,000, including retstart paging and the truncation case described in the issue. Done means requested results are not silently lost when available, or the response clearly reports Count versus the retrieved IDs.