apache / apache/paimon

[Feature] Optimize read table with limit.

Open
#6,847 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

一、Problem Background
In our company's Flink Session-based OLAP environment, we observed that a very basic Flink SQL query against a Paimon table with a LIMIT clause took an unexpectedly long time—over 20 seconds—to complete.
(1)The query is as follows:
`select * from `paimon_na61`.`sample`.`s_holo_mainse_rank_xfc_all_features_swift_parsed` limit 10;`

(2)The table location is :
http://bs.alibaba-inc.com/fsutil?path=dfs%3A//na61dfsalake1--cn-zhangjiakou/alake/omega_na61/sample.db/s_holo_mainse_rank_xfc_all_features_swift_parsed/&isdir=true

二、Root Cause Analysis
Through code review and by attaching to the live Java process with Arthas, we identified the following:
1、The primary bottleneck is the SnapshotReaderImpl#generateSplits method.
(1)We only need select 10 record, but the List is too large.
Image

Image

We found the number of files observed in the manifests system table is also consistent with this large scale.
Image

(2)The main time cost:

http://ha3.oss-cn-hangzhou-zmf.aliyuncs.com/052894/paimon_sample_table_slow.arthas.html

Image

2、Upon further investigation, we found that when executing a SELECT query, the current implementation of AbstractFileStoreScan#plan does not take the LIMIT clause from the SQL into account. This can cause the splits variable (List) within SnapshotReaderImpl#read() to become excessively large. Consequently, when the table consists of a large number of files, SnapshotReaderImpl#generateSplits takes a very long time to execute.

三、Conclusion and Plan
To resolve this, we propose to push the LIMIT clause down into the AbstractFileStoreScan logic.
We plan to create two separate Pull Requests (PRs) for this effort: one to support Append-only tables and another for Primary Key (PK) tables. Since the logic for PK tables is significantly more complex than for Append-only tables, we will implement support for Append-only tables first.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with AbstractFileStoreScan#plan and follow how SnapshotReaderImpl#read invokes generateSplits. Limit the first implementation to append-only tables, as proposed, and trace the Flink SQL LIMIT path into scan planning. Done means an append-only LIMIT query avoids generating an unnecessarily large split list while preserving correct results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.