apache / apache/paimon

[Bug] Flink jobmanager may OOM when the consumption speed cannot keep up with the production speed of the upstream table.

Open
#1,458 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

0.5-snapshot

### Compute Engine

flink

### Minimal reproduce step

Use a `backpressured` flink job to consume paimon, and configure a smaller memory.

### What doesn't meet your expectations?

The `ContinuousFileSplitEnumerator` always scan the newly splits at regular intervals and adds them to the memory queue.

However, when the job is under severe backpressure, the consumption speed will not keep up with the scan speed, which may cause `jobmanager` OOM.

I think the pending split should be limited in `ContinuousFileSplitEnumerator` to avoid OOM.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the ContinuousFileSplitEnumerator and inspect how it scans newly created splits and stores pending splits. Reproduce the issue with a backpressured Flink job and smaller memory settings, then verify that pending splits are bounded and the JobManager no longer runs out of memory under severe backpressure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.