apache / apache/pinot

`order by <sorted column> DESC limit N` queries are not optimised at segment level

Open
#8,837 19 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

A query like

```sql
SELECT *
FROM table
WHERE filter
ORDER BY sorted_col DESC
LIMIT 10;
```

is not optimised and hits the generic `SelectionOrderByOperator.computePartiallyOrdered()` which inserts up to 10k elements into a priority queue to discard all but the last 10. For ascending order, the query below is roughly 3-4x faster per segment.

```sql
SELECT *
FROM table
WHERE filter
ORDER BY sorted_col ASC
LIMIT 10;
```

Contributor guide

Open the contributing guide

Research direction

Start with SelectionOrderByOperator.computePartiallyOrdered() and reproduce the SQL examples for descending and ascending order. Compare their per-segment behavior and establish that the descending query avoids the generic priority-queue path while preserving the requested LIMIT 10 results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.