[Improvement] Add configuration to stop parsing PDFs after X pages
- Dominant language
- Java
- Stars
- 995
- Forks
- 292
- Avg merge
- 2d 49m
- Merged PRs (30d)
- 62
Description
### What would you like to be improved?
Every so often an open crawl will stumble upon a very large pdf, these can take a lot of CPU to parse when effectively most of the content will not be indexed.
### How should we improve?
https://github.com/apache/tika/pull/2803 introduced a config for PDF parsing in Tika to stop processing after X pages. We should make use of it as soon as the next version of Tika is released (currently 3.3.0)
Contributor guide
Research direction
Start by locating StormCrawler’s PDF parsing configuration and the Apache Tika dependency, then check how the crawler passes parser settings. Use the Tika 3.3.0 configuration described in the linked pull request to add a page limit, and verify that large PDFs stop parsing after the configured number of pages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100