apache / apache/lucene

Ability to store Reader / InputStream fields [LUCENE-1206]

Open
#2,283 2 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:core/index type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In some situations we would like to store the whole text, but the whole text won't always fit in memory so we can't create a String. Likewise for storing binary, it would sometimes be better if we didn't have to read into a byte[] up-front (even when it doesn't use much memory, it increases the number of copies made and adds burden to GC.)

FieldsWriter currently writes the length at the start of the chunks though, so I don't know whether it would be possible to seek back and write the length after writing the data.

It would also be useful to use this in conjunction with compression, both for Reader and InputStream types. And when retrieving the field, it should be possible to create a Reader without reading the entire String into memory up-front.

---
Migrated from [LUCENE-1206](https://issues.apache.org/jira/browse/LUCENE-1206) by Trejkaz, updated Mar 02 2013

Contributor guide

Open the contributing guide

Research direction

Start with FieldsWriter, then trace the existing field storage and retrieval paths mentioned in the issue. Determine how Reader and InputStream values could support chunk length handling, compression, and streaming retrieval without loading all content into memory. Done should include a defined implementation and tests covering text and binary storage and retrieval.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.