Avoid timeouts when using a large offset limit
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Follow up from https://apache-pinot.slack.com/archives/C011C9JHN7R/p1621587669055200
---------------
We're having issue with LIMIT on a table with 4.5 millions of rows, when I’m doing this query:
```sql
SELECT * FROM datasource_609bc4f74e3c000300131110 ORDER BY "timestamp" ASC LIMIT 100000,10
```
I’m getting a result in ~2.5s, and I can see in the query response stats totalDocs=4794306 which is fine.
But when I’m doing this one (offset 1 000 000 instead of 100 000):
```sql
SELECT * FROM datasource_609bc4f74e3c000300131110 ORDER BY "timestamp" ASC LIMIT 1000000,10
```
I’m getting no rows in ~10s and the totalDocs is 569840 because Pinot servers timeouts:

We have an hybrid table with segmentPruning by time, we have 16G of heap and around 32G available for nmaped file on each machine. Our segments contains around ~390k documents.
How can we solve this issue without increasing timeouts?
Thank you
Contributor guide
Research direction
Start by reproducing the two SQL queries against a comparable hybrid table and compare their response stats, timeout behavior, and segment-pruning results. Investigate how Pinot handles large OFFSET values across its servers; done means the 1,000,000-offset query returns the expected rows without increasing timeouts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100