apache / apache/paimon

[Bug] OOM when flink query sql insert into a table from another in streaming mode

Open
#2,684 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

0.6.0

### Compute Engine

flink 1.17.1

### Minimal reproduce step

```sql
CREATE TABLE db_issue.ods_order_history
(
field1 INT,
field2 STRING,
field3 STRING,
field4 STRING,
field5 STRING,
field6 STRING,
field7 STRING,
field8 STRING,
field8 INT,
field10 INT,
field11 STRING,
field12 STRING,
field13 STRING,
field14 STRING,
field15 STRING,
field16 STRING,
field17 STRING,
field18 STRING,
field19 BIGINT,
payment_time TIMESTAMP comment 'payment finish time',
data_time TIMESTAMP comment 'data report time',
PRIMARY KEY (field1, field2, field3, field4, field11, field5, field7, field16) NOT ENFORCED
) WITH ('bucket'='8')
;

CREATE TABLE db_issue.ods_order_daily
(
field1 INT,
field2 STRING,
field3 STRING,
field4 STRING,
field5 STRING,
field6 STRING,
field7 STRING,
field8 STRING,
field8 INT,
field10 INT,
field11 STRING,
field12 STRING,
field13 STRING,
field14 STRING,
field15 STRING,
field16 STRING,
field17 STRING,
field18 STRING,
field19 BIGINT,
payment_time TIMESTAMP comment 'payment finish time',
payment_date DATE comment 'payment finish date',
data_time TIMESTAMP comment 'data report time',
PRIMARY KEY (payment_date, field1, field2, field3, field4, field11, field5, field7, field16) NOT ENFORCED
)
PARTITIONED BY (payment_date)
;

set parallelism.default=8;

INSERT INTO ods_order_daily
SELECT
field1,
field2,
field3,
field4,
field5,
field6,
field7,
field8,
field8,
field10,
field11,
field12,
field13,
field14,
field15,
field16,
field17,
field18,
field19,
payment_time,
TO_DATE(DATE_FORMAT(payment_time, 'yyyy-MM-dd')) AS payment_date,
data_time
FROM ods_order_history;
```

### What doesn't meet your expectations?

The ods_order_history table has about 7,000,000 rows, when i run the last sql to insert into ods_order_daily in streaming mode, it cause OOM and restart in loop, but when i set the query in batch mode, i work well
here is the exception logs
image

### Anything else?

None

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the supplied Flink 1.17.1 SQL reproduction and compare the streaming and batch execution paths for the two Paimon tables. Investigate the attached OOM logs and determine why the roughly 7-million-row streaming insert repeatedly exhausts memory; done means the streaming query completes without OOM or restart loops.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.