Ability to store Reader / InputStream fields [LUCENE-1206]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
In some situations we would like to store the whole text, but the whole text won't always fit in memory so we can't create a String. Likewise for storing binary, it would sometimes be better if we didn't have to read into a byte[] up-front (even when it doesn't use much memory, it increases the number of copies made and adds burden to GC.)
FieldsWriter currently writes the length at the start of the chunks though, so I don't know whether it would be possible to seek back and write the length after writing the data.
It would also be useful to use this in conjunction with compression, both for Reader and InputStream types. And when retrieving the field, it should be possible to create a Reader without reading the entire String into memory up-front.
---
Migrated from [LUCENE-1206](https://issues.apache.org/jira/browse/LUCENE-1206) by Trejkaz, updated Mar 02 2013
Contributor guide
Research direction
Start with FieldsWriter, then trace the existing field storage and retrieval paths mentioned in the issue. Determine how Reader and InputStream values could support chunk length handling, compression, and streaming retrieval without loading all content into memory. Done should include a defined implementation and tests covering text and binary storage and retrieval.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100