apache / apache/lucene

Optimization for querying against an index sorted in the opposite order

Open
#15,473 1 comment 1 reaction 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

If an index with very large doc count, is sorted: `myfield:asc`, and the query has sort spec: `myfield:asc`, we can get early-termination goodness and opens doors for many other optimization opportunities.

However, when query has sort spec: `myfield:desc`, this presents to be a worst case scenario because we would be `up-heap` and `rebalance` for every hit, because larger docids will always win the competition against the min-val of the priority queue.

The proposal is to catch this case, and use a FIFO queue like `ArrayDequeue` instead of `PriorityQueue` to always insert into the queue.

I am not sure if this is too much of an edge case to be included in Lucene, feel free to close the issue if so.

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by tracing the search path that uses PriorityQueue for an index sorted opposite to the query sort, then assess the proposed FIFO queue approach for this case. Done should mean the opposite-order query avoids per-hit heap rebalancing without changing its results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.