apache / apache/pinot

Avoid timeouts when using a large offset limit

Open
#6,982 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Follow up from https://apache-pinot.slack.com/archives/C011C9JHN7R/p1621587669055200

---------------

We're having issue with LIMIT on a table with 4.5 millions of rows, when I’m doing this query:
```sql
SELECT * FROM datasource_609bc4f74e3c000300131110 ORDER BY "timestamp" ASC LIMIT 100000,10
```
I’m getting a result in ~2.5s, and I can see in the query response stats totalDocs=4794306 which is fine.

But when I’m doing this one (offset 1 000 000 instead of 100 000):
```sql
SELECT * FROM datasource_609bc4f74e3c000300131110 ORDER BY "timestamp" ASC LIMIT 1000000,10
```
I’m getting no rows in ~10s and the totalDocs is 569840 because Pinot servers timeouts:
Capture d’écran 2021-05-25 à 10 00 34

We have an hybrid table with segmentPruning by time, we have 16G of heap and around 32G available for nmaped file on each machine. Our segments contains around ~390k documents.

How can we solve this issue without increasing timeouts?

Thank you

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two SQL queries against a comparable hybrid table and compare their response stats, timeout behavior, and segment-pruning results. Investigate how Pinot handles large OFFSET values across its servers; done means the 1,000,000-offset query returns the expected rows without increasing timeouts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.