apache / apache/paimon

[Bug] Flink S3 cause TaskManager Metaspace OOM

Open
#2,780 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

Paimon 0.6.0

### Compute Engine

Flink1.18.0

### Minimal reproduce step

1.use flink sql-client.sh
2.not move paimon jar into flink/lib
3.execute follow sql many times

create table Orders (
order_number bigint,
price decimal(32,2),
order_time timestamp(3)
) with (
'connector' = 'datagen'
);

add jar 'C:\flink_lib\hadoop-mapreduce-client-core-2.10.1.jar';
add jar 'C:\flink_lib\hadoop-hdfs-client-2.8.5.jar';
add jar 'C:\flink_lib\paimon-flink-1.18-0.6.0-incubating.jar';
add jar 'C:\flink_lib\paimon-s3-0.6.0-incubating.jar';

create table paimon_order (
order_number bigint,
price decimal(32,2),
order_time timestamp(3)
) with (
'connector' = 'paimon',
'path' = 's3://local-test/paimon_order/',
's3.endpoint' = 'http://127.0.0.1:9000/',
's3.access-key' = '',
's3.secret-key' = '',
'bucket' = '1',
'auto-create'= 'true'
);

insert into paimon_order select order_number, price, order_time from Orders;

### What doesn't meet your expectations?

now, execute it once, metaspace use more memory, util oom.
I find it maybe caused by S3FileIO, paimon just create S3AFileSystem, but not close it.
thread dump we can see s3a-file thread and java-sdk-http-connection-reaper thread, this all create by new S3AFileSystem().initialize().
s3a-file-thread can set env.java.opts.taskmanager: -Dhadoop.metrics.init.mode=STANDBY to avoid init,
but java-sdk-http-connection-reaper thread has no config can avoid init it.
both of threads can't finish when task finish or crash or cancel, it caused childFirstClassLoader can't release memory.
and i see S3AFileSystem provide close method to close both thread, but i can't get it to execute

so any idea to fix it. by the way, I'm not sure if executing the close method will fix this issue, But now Mat analyzes the current impact of this

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with Flink 1.18's sql-client.sh and the repeated SQL sequence described, then inspect the thread dump around S3FileIO and S3AFileSystem.initialize(). Trace how the java-sdk-http-connection-reaper and s3a-file threads are created and whether S3AFileSystem.close() is reachable when the task finishes, crashes, or is cancelled. Done means the resource lifecycle is handled and repeated execution no longer causes TaskManager Metaspace OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.