bytedance / bytedance/pdf-parser

perf: avoid an unconditional full-document scan for narrow page ranges

Open
#21 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
14
Forks
9
Avg merge
43m
Merged PRs (30d)
1

Description

## Problem

A narrow `page_range` limits block extraction but does not limit the earlier header/footer similarity pass. With the default `skip_header_footer=True`, parsing one requested page still calls `get_text("dict")` across every page in the document before extracting the selected page.

This affects current `main` at `c6478aae9ab50a13fc54c0599a0a459396c15b99` and release `0.0.5`.

## Source evidence

In `PyMuPDFParser.parse()`:

1. `_get_similarity_blocks(document)` runs before `page_range` is passed to `_extract_all_blocks()`.
2. `_get_similarity_blocks()` executes `for page in document` and calls `page.get_text("dict")` for each page.
3. Only afterward does `_extract_all_blocks()` resolve the requested start/end page indices.

Thus `page_range=(500, 500)` on a 1,000-page PDF performs the similarity scan for all 1,000 pages and then extracts page 500.

## Expected behavior

A small requested range should have a bounded or explicitly documented cost. If whole-document sampling is required for header/footer accuracy, the parser could use a bounded sample, expose a documented tradeoff, or allow callers to disable that pass for range-limited work.

## Impact

CLI and HTTP callers requesting a small subset of a large document can still pay most of the text-extraction cost and memory churn of scanning the complete PDF.

## Why there is no PR in this report

Changing the similarity pass affects header/footer detection accuracy and parser performance semantics. A safe implementation needs maintainer direction on sampling, configuration behavior, and acceptable accuracy, followed by benchmarks on representative large PDFs that are not present in the repository fixtures.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in PyMuPDFParser.parse(), tracing _get_similarity_blocks() and _extract_all_blocks() with a narrow page_range. Review the current full-document get_text("dict") pass and determine, with maintainer guidance, whether sampling, configuration, or bounded scanning preserves acceptable header/footer accuracy. Validate the chosen behavior with benchmarks on representative large PDFs; repository fixtures do not include those benchmarks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.