bytedance / bytedance/pdf-parser
perf: avoid an unconditional full-document scan for narrow page ranges
- Dominant language
- Python
- Stars
- 14
- Forks
- 9
- Avg merge
- 43m
- Merged PRs (30d)
- 1
Description
## Problem
A narrow `page_range` limits block extraction but does not limit the earlier header/footer similarity pass. With the default `skip_header_footer=True`, parsing one requested page still calls `get_text("dict")` across every page in the document before extracting the selected page.
This affects current `main` at `c6478aae9ab50a13fc54c0599a0a459396c15b99` and release `0.0.5`.
## Source evidence
In `PyMuPDFParser.parse()`:
1. `_get_similarity_blocks(document)` runs before `page_range` is passed to `_extract_all_blocks()`.
2. `_get_similarity_blocks()` executes `for page in document` and calls `page.get_text("dict")` for each page.
3. Only afterward does `_extract_all_blocks()` resolve the requested start/end page indices.
Thus `page_range=(500, 500)` on a 1,000-page PDF performs the similarity scan for all 1,000 pages and then extracts page 500.
## Expected behavior
A small requested range should have a bounded or explicitly documented cost. If whole-document sampling is required for header/footer accuracy, the parser could use a bounded sample, expose a documented tradeoff, or allow callers to disable that pass for range-limited work.
## Impact
CLI and HTTP callers requesting a small subset of a large document can still pay most of the text-extraction cost and memory churn of scanning the complete PDF.
## Why there is no PR in this report
Changing the similarity pass affects header/footer detection accuracy and parser performance semantics. A safe implementation needs maintainer direction on sampling, configuration behavior, and acceptable accuracy, followed by benchmarks on representative large PDFs that are not present in the repository fixtures.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in PyMuPDFParser.parse(), tracing _get_similarity_blocks() and _extract_all_blocks() with a narrow page_range. Review the current full-document get_text("dict") pass and determine, with maintainer guidance, whether sampling, configuration, or bounded scanning preserves acceptable header/footer accuracy. Validate the chosen behavior with benchmarks on representative large PDFs; repository fixtures do not include those benchmarks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100