apache / apache/pinot

Using DocValues Instead of Stored Fields

Open
#7,217 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

When creating text indices, we use Lucene's stored fields to store the DocID field inside the index. This is inefficient due stored fields' representation within Lucene.

A more efficient way would be use Lucene's DocValues and allow DocValues to be used as stored fields. This will reduce our storage footprint.

Contributor guide

Open the contributing guide

Research direction

Start by locating Pinot's text-index creation and DocID stored-field handling, then review how Lucene DocValues could replace that representation. Done means the DocID uses DocValues as stored data and the resulting text-index storage footprint is reduced.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.