apache / apache/lucene

Can we use skipper information to improve writing/merging of doc_values fields?

Open
#15,884 2 comments 2 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

When Lucene90DocValuesConsumer writes a numeric doc values field, it iterates over the values multiple times - once to write skippers if they are configured, once to get minimum and maximum values for block encoding, once to write the values. SortedNumeric repeats this, and then iterates again to write addresses if there are documents with multiple values. Sorted and SortedSet re-use the numeric values machinery to write their ords, and SortedSet also adds in an iteration over all values to check to see if it can fall back to just Sorted. BinaryDocValues iterates over everything to get minimum and maximum lengths before writing values out.

We should be able to use skipper metadata (where it exists) to avoid at least some of these multiple iterations, especially if we add more information such as maxValuesPerDoc.

Contributor guide

Open the contributing guide

Research direction

Start with Lucene90DocValuesConsumer and the numeric, SortedNumeric, Sorted, SortedSet, and BinaryDocValues write paths named in the issue. Examine how skipper metadata is configured and used, then run the relevant existing doc-values tests while determining how additional metadata such as maxValuesPerDoc could reduce repeated iteration without changing output.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.