apache / apache/paimon

[Bug] HadoopFileIO does not close per-instance FileSystems, causing executor thread growth with disabled Hadoop FS cache

Open
#8,548 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

- Paimon: 1.1.1 in the affected deployment
- Hadoop FileSystem: Aliyun OSS Hadoop FileSystem
- Access pattern: many concurrent Paimon scan splits
- Hadoop FS cache: disabled to isolate credentials

### Compute Engine

java

### Minimal reproduce step

We observed unbounded JVM thread growth when Paimon reads object storage through `HadoopFileIO` in an embedded compute-engine JVM.

The issue is reproducible when all of the following are true:

1. The engine creates/deserializes a Paimon `Table` (and thus a `HadoopFileIO`) for many scan splits.
2. Hadoop FileSystem cache is disabled, for example:

```text
fs.oss.impl.disable.cache=true

This configuration is commonly required to avoid credential reuse between different AK/SK or STS tokens.
3. The underlying Hadoop FileSystem implementation creates executor threads. In our case it is:

org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem

## Observed behavior

HadoopFileIO maintains a per-instance map:

Map, FileSystem> fsMap;

When the Hadoop global FileSystem cache is disabled, each HadoopFileIO instance creates its own FileSystem instance through
path.getFileSystem(conf).

However, HadoopFileIO does not override close(). Therefore, calling:

table.fileIO().close();

does not close the FileSystem instances because FileIO.close() is currently a no-op by default.

For OSS, each unclosed AliyunOSSFileSystem may retain transfer/copy executor threads. With many scan splits, the JVM thread count keeps
increasing and can eventually reach thousands or more.

We checked Paimon 1.4.2, the release-1.4 branch, and current master; HadoopFileIO still has fsMap but does not close its FileSystem
instances.

### What doesn't meet your expectations?

When a HadoopFileIO instance is no longer used, its owned non-shared FileSystem instances should be released so that connector-specific
resources, including executor threads, are shut down.

## Question / proposal

Would the Paimon community consider implementing resource cleanup in HadoopFileIO.close()?

Conceptually, it would close and clear the FileSystem instances stored in the instance-local fsMap.

There are two design questions we would appreciate guidance on:

1. Is HadoopFileIO intended to own and close the FileSystem instances in its fsMap?
2. How should cleanup behave when Hadoop global FileSystem cache is enabled, so that closing one HadoopFileIO does not unexpectedly affect
another user of a shared cached FileSystem?

A Paimon-level fix would help all compute engines using HadoopFileIO with disabled Hadoop FS cache, not only our embedded-engine scenario.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HadoopFileIO and its instance-local fsMap, then trace how path.getFileSystem(conf) creates and caches each FileSystem and how FileIO.close() is currently handled. Review the shared-cache behavior and existing resource-lifecycle tests, if present. Done means owned non-shared resources are released without unexpectedly closing shared cached FileSystem instances.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.