Using DocValues Instead of Stored Fields
Open
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
When creating text indices, we use Lucene's stored fields to store the DocID field inside the index. This is inefficient due stored fields' representation within Lucene.
A more efficient way would be use Lucene's DocValues and allow DocValues to be used as stored fields. This will reduce our storage footprint.
Contributor guide
Research direction
Start by locating Pinot's text-index creation and DocID stored-field handling, then review how Lucene DocValues could replace that representation. Done means the DocID uses DocValues as stored data and the resulting text-index storage footprint is reduced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100