apache / apache/hudi

[SUPPORT] OOM occurs a few days after the Flink jobmanager restarts

Open
#7,407 10 comments 0 reactions 0 assignees View on GitHub
engine:flink priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

A flink job writes to hudi table , but the jobmanager always has OOM after a period of time.
```
Application application_1669378398064_0078 failed 1 times (global limit =2; local limit is =1) due to AM Container for
appattempt_1669378398064_0078_000001 exited with exitCode: -104

Failing this attempt.Diagnostics: [2022-12-08 13:19:49.156]Container
[pid=39129,containerID=container_e07_1669378398064_0078_01_000001] is running beyond physical memory limits. Current usage:
4.0 GB of 4 GB physical memory used; 7.0 GB of 8.4 GB virtual memory used. Killing container.
```

my table options:
```
'read.streaming.enabled' = 'true',
'path' = 'hdfs://xxx/hudi-warehouse/mytable',
'hive_sync.enable' = 'true',
'connector' = 'hudi',
'read.streaming.check-interval' = '30',
'hoodie.datasource.write.hive_style_partitioning' = 'true',
'index.state.ttl' = '3650',
'hoodie.datasource.write.partitionpath.field' = 'create_day',
'changelog.enabled' = 'true',
'clean.retain_commits' = '300',
'table.type' = 'MERGE_ON_READ',
'hive_sync.mode' = 'hms',

```

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.11.1

* Flink version : 0.14.5

**My Questions**

1. Is this normal?
2. Why does jobmanager use so much memory?
3. Should I increase memory or use other tuning methods

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reported Hudi 0.11.1 and Flink 0.14.5 environment, the JobManager OOM diagnostics, and the listed table options. Reproduce or investigate memory growth after a restart and determine whether the result is expected, requires more memory, or needs a tuning change; the issue is done when the cause and supported remediation are documented.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.