apache / apache/lucene

fix slow uses of SortedDocValues in join/ and misc/ [LUCENE-9841]

Open
#10,880 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Background: #10835, #10834

While fixing the API, I discovered some bad guys doing lookupOrd() on every document. But we just fixed the API in #10835 and didn't fix the slow things these two are doing yet.

It may be a good idea to break these two problems down into separate subtasks:

- lucene/join: This has some slow implementations exposed, but fast ones are available (e.g. using SortedSetDocValues). Seems really easy to fix: simply use the SortedSetDocValues algorithm for SortedDocValues too, and remove the slow stuff.
- lucene/misc: DocValuesStats seems to just want the min and max values for Sorted and SortedSet. This is can be efficiently done with ordinals instead of bytes: e.g. just get the min and max ordinal for each segment and lookupOrd twice at the end of processing the segment.

---
Migrated from [LUCENE-9841](https://issues.apache.org/jira/browse/LUCENE-9841) by Robert Muir (@rmuir)

Contributor guide

Open the contributing guide

Research direction

Read the context in #10835 and #10834, then inspect the SortedDocValues uses in lucene/join and lucene/misc. In lucene/join, compare the SortedDocValues path with the SortedSetDocValues algorithm; in lucene/misc, inspect DocValuesStats. Done means avoiding per-document lookupOrd() while preserving the intended statistics and join behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.