fix slow uses of SortedDocValues in join/ and misc/ [LUCENE-9841]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Background: #10835, #10834
While fixing the API, I discovered some bad guys doing lookupOrd() on every document. But we just fixed the API in #10835 and didn't fix the slow things these two are doing yet.
It may be a good idea to break these two problems down into separate subtasks:
- lucene/join: This has some slow implementations exposed, but fast ones are available (e.g. using SortedSetDocValues). Seems really easy to fix: simply use the SortedSetDocValues algorithm for SortedDocValues too, and remove the slow stuff.
- lucene/misc: DocValuesStats seems to just want the min and max values for Sorted and SortedSet. This is can be efficiently done with ordinals instead of bytes: e.g. just get the min and max ordinal for each segment and lookupOrd twice at the end of processing the segment.
---
Migrated from [LUCENE-9841](https://issues.apache.org/jira/browse/LUCENE-9841) by Robert Muir (@rmuir)
Contributor guide
Research direction
Read the context in #10835 and #10834, then inspect the SortedDocValues uses in lucene/join and lucene/misc. In lucene/join, compare the SortedDocValues path with the SortedSetDocValues algorithm; in lucene/misc, inspect DocValuesStats. Done means avoiding per-document lookupOrd() while preserving the intended statistics and join behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100