[Bug] Symptom oom is displayed when hive reads data using limit
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
Scenario description:
1. A partition contains 17 million data
2. 1 bucket
3. Query script: select * from dwd.paimon_table_test where dt='20211231' limit 10;
4.Number of files in the partition: 583 Average file size: 2.6mb
Abnormal information:
When the limit command is used to query data, fetch cannot be used. Data is read directly through mapreduce
### Compute Engine
hive :cdh-6.3.2 hive 2.1.1
Paimon: master branches
### Minimal reproduce step
A large amount of data is written into the paimon table from the hive table to generate multiple small files
### What doesn't meet your expectations?
The limit operation should not cause oom, and the limit operation can fetch directly
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the query against the large partition with Hive 2.1.1 on CDH 6.3.2 and Paimon master, using the reported 17 million rows and 583 files. Compare the limited query's memory behavior with fetch-based reading; done means LIMIT 10 completes without OOM and can use fetch directly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100