apache / apache/lucene

Specialize write path for sorted doc values [LUCENE-10133]

Open
#11,170 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

#10653 caused an indexing slowdown, but we should be able to mitigate it while still using the same on-disk representation for ordinals and numerics by specializing the write path for sorted doc values. For instance, there is no need to recompute the min and max values for sorted doc values, since sorted doc values already have those precomputed.

---
Migrated from [LUCENE-10133](https://issues.apache.org/jira/browse/LUCENE-10133) by Adrien Grand (@jpountz), updated Oct 05 2021
Pull requests: https://github.com/apache/lucene/pull/330, https://github.com/apache/lucene/pull/330

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the regression described in #10653 and the linked pull request #330, then trace the sorted doc values write path. Done means specializing that path while preserving the existing on-disk representation and avoiding unnecessary min/max recomputation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.