[Bug] Flink S3 cause TaskManager Metaspace OOM
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
Paimon 0.6.0
### Compute Engine
Flink1.18.0
### Minimal reproduce step
1.use flink sql-client.sh
2.not move paimon jar into flink/lib
3.execute follow sql many times
create table Orders (
order_number bigint,
price decimal(32,2),
order_time timestamp(3)
) with (
'connector' = 'datagen'
);
add jar 'C:\flink_lib\hadoop-mapreduce-client-core-2.10.1.jar';
add jar 'C:\flink_lib\hadoop-hdfs-client-2.8.5.jar';
add jar 'C:\flink_lib\paimon-flink-1.18-0.6.0-incubating.jar';
add jar 'C:\flink_lib\paimon-s3-0.6.0-incubating.jar';
create table paimon_order (
order_number bigint,
price decimal(32,2),
order_time timestamp(3)
) with (
'connector' = 'paimon',
'path' = 's3://local-test/paimon_order/',
's3.endpoint' = 'http://127.0.0.1:9000/',
's3.access-key' = '',
's3.secret-key' = '',
'bucket' = '1',
'auto-create'= 'true'
);
insert into paimon_order select order_number, price, order_time from Orders;
### What doesn't meet your expectations?
now, execute it once, metaspace use more memory, util oom.
I find it maybe caused by S3FileIO, paimon just create S3AFileSystem, but not close it.
thread dump we can see s3a-file thread and java-sdk-http-connection-reaper thread, this all create by new S3AFileSystem().initialize().
s3a-file-thread can set env.java.opts.taskmanager: -Dhadoop.metrics.init.mode=STANDBY to avoid init,
but java-sdk-http-connection-reaper thread has no config can avoid init it.
both of threads can't finish when task finish or crash or cancel, it caused childFirstClassLoader can't release memory.
and i see S3AFileSystem provide close method to close both thread, but i can't get it to execute
so any idea to fix it. by the way, I'm not sure if executing the close method will fix this issue, But now Mat analyzes the current impact of this
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the issue with Flink 1.18's sql-client.sh and the repeated SQL sequence described, then inspect the thread dump around S3FileIO and S3AFileSystem.initialize(). Trace how the java-sdk-http-connection-reaper and s3a-file threads are created and whether S3AFileSystem.close() is reachable when the task finishes, crashes, or is cancelled. Done means the resource lifecycle is handled and repeated execution no longer causes TaskManager Metaspace OOM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100