apache / apache/stormcrawler

[Improvement] Add configuration to stop parsing PDFs after X pages

Open
#1,901 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
995
Forks
292
Avg merge
2d 49m
Merged PRs (30d)
62

Description

### What would you like to be improved?

Every so often an open crawl will stumble upon a very large pdf, these can take a lot of CPU to parse when effectively most of the content will not be indexed.

### How should we improve?

https://github.com/apache/tika/pull/2803 introduced a config for PDF parsing in Tika to stop processing after X pages. We should make use of it as soon as the next version of Tika is released (currently 3.3.0)

Contributor guide

Open the contributing guide

Research direction

Start by locating StormCrawler’s PDF parsing configuration and the Apache Tika dependency, then check how the crawler passes parser settings. Use the Tika 3.3.0 configuration described in the linked pull request to add a page limit, and verify that large PDFs stop parsing after the configured number of pages.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.