Can IndexingChain better protect against large documents? [LUCENE-9994]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
It's easy for a single document to use several times the amount of RAM that is configured on IndexWriter by having many fields or many terms on a single field. Could we improve IndexingChain to reject such documents before they may cause an out-of-memory error? We could make such documents born deleted in the new segment like we already do when consuming a TokenStream raises an exception.
---
Migrated from [LUCENE-9994](https://issues.apache.org/jira/browse/LUCENE-9994) by Adrien Grand (@jpountz)
Contributor guide
Research direction
Start by locating IndexingChain and the existing handling for documents where consuming a TokenStream raises an exception. Determine how oversized documents should be detected and represented as born deleted, then add coverage showing that large documents are rejected without causing an out-of-memory error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100