Can we use skipper information to improve writing/merging of doc_values fields?
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
When Lucene90DocValuesConsumer writes a numeric doc values field, it iterates over the values multiple times - once to write skippers if they are configured, once to get minimum and maximum values for block encoding, once to write the values. SortedNumeric repeats this, and then iterates again to write addresses if there are documents with multiple values. Sorted and SortedSet re-use the numeric values machinery to write their ords, and SortedSet also adds in an iteration over all values to check to see if it can fall back to just Sorted. BinaryDocValues iterates over everything to get minimum and maximum lengths before writing values out.
We should be able to use skipper metadata (where it exists) to avoid at least some of these multiple iterations, especially if we add more information such as maxValuesPerDoc.
Contributor guide
Research direction
Start with Lucene90DocValuesConsumer and the numeric, SortedNumeric, Sorted, SortedSet, and BinaryDocValues write paths named in the issue. Examine how skipper metadata is configured and used, then run the relevant existing doc-values tests while determining how additional metadata such as maxValuesPerDoc could reduce repeated iteration without changing output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100