Explore a combined DocValues/PostingsFormat that shares a single terms dict [LUCENE-5832]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
With #6881, which adds ordinals support to block tree terms dict, it's in theory possible to create a codec that uses a single shared terms dict for both Sorted/SetDocValues and postings as long as the app "promises" that the given field(s) will always be indexed as both S/SDV and inverted.
The codec should ideally throw an exc if that promise was broken, instead of adding incorrect values to the index.
This would save RAM because at search time only one shared "terms index" is loaded, vs two today. Of course the forward index for S/SDV (mapping doc -> term ords) must still be separately stored from the inverted postings (mapping term ord -> docs).
---
Migrated from [LUCENE-5832](https://issues.apache.org/jira/browse/LUCENE-5832) by Michael McCandless (@mikemccand), 2 votes
Contributor guide
Research direction
Start by reviewing #6881 and the migrated LUCENE-5832 discussion to understand the existing block tree terms dictionary and ordinals support. Determine the codec entry points and tests involved before defining the shared terms dictionary design. Done means a codec can share one terms dictionary for postings and Sorted/SetDocValues while rejecting fields that violate the stated indexing promise.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100