[Bug] OOM when flink query sql insert into a table from another in streaming mode
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
0.6.0
### Compute Engine
flink 1.17.1
### Minimal reproduce step
```sql
CREATE TABLE db_issue.ods_order_history
(
field1 INT,
field2 STRING,
field3 STRING,
field4 STRING,
field5 STRING,
field6 STRING,
field7 STRING,
field8 STRING,
field8 INT,
field10 INT,
field11 STRING,
field12 STRING,
field13 STRING,
field14 STRING,
field15 STRING,
field16 STRING,
field17 STRING,
field18 STRING,
field19 BIGINT,
payment_time TIMESTAMP comment 'payment finish time',
data_time TIMESTAMP comment 'data report time',
PRIMARY KEY (field1, field2, field3, field4, field11, field5, field7, field16) NOT ENFORCED
) WITH ('bucket'='8')
;
CREATE TABLE db_issue.ods_order_daily
(
field1 INT,
field2 STRING,
field3 STRING,
field4 STRING,
field5 STRING,
field6 STRING,
field7 STRING,
field8 STRING,
field8 INT,
field10 INT,
field11 STRING,
field12 STRING,
field13 STRING,
field14 STRING,
field15 STRING,
field16 STRING,
field17 STRING,
field18 STRING,
field19 BIGINT,
payment_time TIMESTAMP comment 'payment finish time',
payment_date DATE comment 'payment finish date',
data_time TIMESTAMP comment 'data report time',
PRIMARY KEY (payment_date, field1, field2, field3, field4, field11, field5, field7, field16) NOT ENFORCED
)
PARTITIONED BY (payment_date)
;
set parallelism.default=8;
INSERT INTO ods_order_daily
SELECT
field1,
field2,
field3,
field4,
field5,
field6,
field7,
field8,
field8,
field10,
field11,
field12,
field13,
field14,
field15,
field16,
field17,
field18,
field19,
payment_time,
TO_DATE(DATE_FORMAT(payment_time, 'yyyy-MM-dd')) AS payment_date,
data_time
FROM ods_order_history;
```
### What doesn't meet your expectations?
The ods_order_history table has about 7,000,000 rows, when i run the last sql to insert into ods_order_daily in streaming mode, it cause OOM and restart in loop, but when i set the query in batch mode, i work well
here is the exception logs
### Anything else?
None
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the supplied Flink 1.17.1 SQL reproduction and compare the streaming and batch execution paths for the two Paimon tables. Investigate the attached OOM logs and determine why the roughly 7-million-row streaming insert repeatedly exhausts memory; done means the streaming query completes without OOM or restart loops.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100