[Bug] HadoopFileIO does not close per-instance FileSystems, causing executor thread growth with disabled Hadoop FS cache
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
- Paimon: 1.1.1 in the affected deployment
- Hadoop FileSystem: Aliyun OSS Hadoop FileSystem
- Access pattern: many concurrent Paimon scan splits
- Hadoop FS cache: disabled to isolate credentials
### Compute Engine
java
### Minimal reproduce step
We observed unbounded JVM thread growth when Paimon reads object storage through `HadoopFileIO` in an embedded compute-engine JVM.
The issue is reproducible when all of the following are true:
1. The engine creates/deserializes a Paimon `Table` (and thus a `HadoopFileIO`) for many scan splits.
2. Hadoop FileSystem cache is disabled, for example:
```text
fs.oss.impl.disable.cache=true
This configuration is commonly required to avoid credential reuse between different AK/SK or STS tokens.
3. The underlying Hadoop FileSystem implementation creates executor threads. In our case it is:
org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
## Observed behavior
HadoopFileIO maintains a per-instance map:
Map, FileSystem> fsMap;
When the Hadoop global FileSystem cache is disabled, each HadoopFileIO instance creates its own FileSystem instance through
path.getFileSystem(conf).
However, HadoopFileIO does not override close(). Therefore, calling:
table.fileIO().close();
does not close the FileSystem instances because FileIO.close() is currently a no-op by default.
For OSS, each unclosed AliyunOSSFileSystem may retain transfer/copy executor threads. With many scan splits, the JVM thread count keeps
increasing and can eventually reach thousands or more.
We checked Paimon 1.4.2, the release-1.4 branch, and current master; HadoopFileIO still has fsMap but does not close its FileSystem
instances.
### What doesn't meet your expectations?
When a HadoopFileIO instance is no longer used, its owned non-shared FileSystem instances should be released so that connector-specific
resources, including executor threads, are shut down.
## Question / proposal
Would the Paimon community consider implementing resource cleanup in HadoopFileIO.close()?
Conceptually, it would close and clear the FileSystem instances stored in the instance-local fsMap.
There are two design questions we would appreciate guidance on:
1. Is HadoopFileIO intended to own and close the FileSystem instances in its fsMap?
2. How should cleanup behave when Hadoop global FileSystem cache is enabled, so that closing one HadoopFileIO does not unexpectedly affect
another user of a shared cached FileSystem?
A Paimon-level fix would help all compute engines using HadoopFileIO with disabled Hadoop FS cache, not only our embedded-engine scenario.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with HadoopFileIO and its instance-local fsMap, then trace how path.getFileSystem(conf) creates and caches each FileSystem and how FileIO.close() is currently handled. Review the shared-cache behavior and existing resource-lifecycle tests, if present. Done means owned non-shared resources are released without unexpectedly closing shared cached FileSystem instances.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100