Ability to turn off the store for an index [LUCENE-2025]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
It would be really good in combination with parallel indexing if the
Lucene store could be turned off entirely for an index.
The reason is that part of the store is the FieldIndex (.fdx file),
which contains an 8 bytes pointer for each document in a segment, even
if a document does not contain any stored fields.
With parallel indexing we will want to rewrite certain parallel
indexes to update them, and if such an update affects only a small
number of documents it will be a waste if you have to write the .fdx
file every time.
So in the case where you only want to update a data structure in the
inverted index it makes sense to separate your index into multiple
parallel indexes, where the ones you want to update don't contain any
stored fields.
It'd be also great to not only allow turning off the store but to make
it customizable, similarly to what flexible indexing wants to achieve
regarding the inverted index.
As a start I'd be happy with the ability to simply turn off the store and to
add more flexibility later.
---
Migrated from [LUCENE-2025](https://issues.apache.org/jira/browse/LUCENE-2025) by Michael Busch, 1 vote, updated May 09 2016
Linked issues:
- #2954
- #2532
Contributor guide
Research direction
Start by reviewing the issue's parallel-indexing rationale and the linked issues #2954 and #2532. Determine how Lucene's store and FieldIndex are currently tied to an index, then define a focused first step for disabling the store while leaving later customization out of scope. Done means an index can omit stored fields without writing the corresponding store data, with relevant behavior covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100