apache / apache/doris

[Bug](fs) DFSFileSystem phantom cleanup may close Hadoop FileSystem while operations are still running

Open
#64,369 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

branch-4.0

### What's Wrong?

In branch-4.0, `RemoteFSPhantomManager` tracks `DFSFileSystem` with a `PhantomReference` and closes the internal `org.apache.hadoop.fs.FileSystem` when the `DFSFileSystem` object is garbage collected.

This can close the underlying Hadoop `FileSystem` too early. Some DFS operations may still be using the Hadoop `FileSystem`, `RemoteIterator`, `FSDataInputStream`, or `FSDataOutputStream` after the owning `DFSFileSystem` object is no longer strongly reachable. In that window, GC can enqueue the `DFSFileSystem` phantom reference, and the cleanup thread may close the Hadoop `FileSystem` while operations such as `listStatus/listFiles`, read, or write are still running.

This may cause intermittent failures in external file system access, especially when:
- `DFSFileSystem` is evicted from `FileSystemCache` by size limit
- `DFSFileSystem` expires after access timeout
- `DFSFileSystem` is created directly and not retained by the cache
- a long-running list/read/write operation only keeps the underlying Hadoop objects alive

### What You Expected?

The internal Hadoop `FileSystem` should not be closed while any active DFS operation is still using it.

Cleanup should happen only after both conditions are true:
1. The owning `DFSFileSystem` is closed or garbage collected.
2. There are no active operations/streams/iterators holding the underlying Hadoop `FileSystem`.

### How to Reproduce?

The issue is timing-sensitive and depends on GC.

General reproduction scenario:

1. Create or obtain a `DFSFileSystem`.
2. Start a long-running HDFS operation, for example recursive `listFiles/listStatus`, or create an input/output stream.
3. Drop the strong reference to the `DFSFileSystem` while the returned Hadoop object is still being used.
4. Trigger GC.
5. Wait for `RemoteFSPhantomManager` cleanup.
6. Continue using the iterator/stream.

The cleanup thread may close the internal Hadoop `FileSystem`, causing the still-running operation to fail with IO errors related to a closed filesystem/client.

This is less likely during normal cache hits because `FileSystemCache` uses a strong-value Caffeine cache, but the race is still possible after cache eviction, expiration, or direct construction outside the cache.

### Anything Else?

The root cause is that phantom cleanup is tied to the lifetime of `DFSFileSystem`, but the actual resource users are the underlying Hadoop `FileSystem` and streams/iterators derived from it.

A safer design is to introduce a resource/lease layer:
- `DFSFileSystemResource` owns the Hadoop `FileSystem`.
- Each operation acquires a lease before using the resource.
- `close()` or phantom cleanup only marks the resource as closing.
- The Hadoop `FileSystem` is physically closed only after all active leases are released.

This avoids closing the Hadoop `FileSystem` while `listFiles`, reads, or writes are still in progress.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.