apache / apache/paimon

[Bug] Symptom oom is displayed when hive reads data using limit

Open
#1,253 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

Scenario description:
1. A partition contains 17 million data
2. 1 bucket
3. Query script: select * from dwd.paimon_table_test where dt='20211231' limit 10;
4.Number of files in the partition: 583 Average file size: 2.6mb

Abnormal information:
image

When the limit command is used to query data, fetch cannot be used. Data is read directly through mapreduce

### Compute Engine

hive :cdh-6.3.2 hive 2.1.1
Paimon: master branches

### Minimal reproduce step

A large amount of data is written into the paimon table from the hive table to generate multiple small files

### What doesn't meet your expectations?

The limit operation should not cause oom, and the limit operation can fetch directly

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the query against the large partition with Hive 2.1.1 on CDH 6.3.2 and Paimon master, using the reported 17 million rows and 583 files. Compare the limited query's memory behavior with fetch-based reading; done means LIMIT 10 completes without OOM and can use fetch directly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.