[Bug] Flink jobmanager may OOM when the consumption speed cannot keep up with the production speed of the upstream table.
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
0.5-snapshot
### Compute Engine
flink
### Minimal reproduce step
Use a `backpressured` flink job to consume paimon, and configure a smaller memory.
### What doesn't meet your expectations?
The `ContinuousFileSplitEnumerator` always scan the newly splits at regular intervals and adds them to the memory queue.
However, when the job is under severe backpressure, the consumption speed will not keep up with the scan speed, which may cause `jobmanager` OOM.
I think the pending split should be limited in `ContinuousFileSplitEnumerator` to avoid OOM.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the ContinuousFileSplitEnumerator and inspect how it scans newly created splits and stores pending splits. Reproduce the issue with a backpressured Flink job and smaller memory settings, then verify that pending splits are bounded and the JobManager no longer runs out of memory under severe backpressure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100