CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.10.0
### Query engine
Spark
### Please describe the bug 🐞
Apache Iceberg version: 1.10.0
Component: org.apache.iceberg.CachingCatalog
Description
CachingCatalog uses a Caffeine cache to hold Table objects. When an entry is evicted (by TTL via cache.expiration-interval-ms or by size via cache.max-total-bytes), the RemovalListener
(MetadataTableInvalidatingRemovalListener) only invalidates related metadata table entries. It does not call table.io().close().
This means any resources held by the FileIO implementation are never released on eviction.
Impact
With io-impl = org.apache.iceberg.aws.s3.S3FileIO:
- Each evicted Table leaves behind a live AWS SDK v2 S3Client
- Each S3Client owns a ScheduledExecutorService (sdk-ScheduledExecutor-N) with background threads for credential refresh (IMDSv2)
- These threads are GC roots — they can never be collected
- In a long-running process (e.g. Spark Thrift Server), threads accumulate without bound until the JVM crashes with os::commit_memory failed; error='Not enough space' (errno=12)
Observed in production (Spark Thrift Server, ~24h uptime):
Total JVM threads: 27,877
sdk-ScheduledExecutor: 27,657
Distinct pool instances: 8,075+
Proof from bytecode
CachingCatalog$MetadataTableInvalidatingRemovalListener.onRemoval() decompiled from iceberg-spark-runtime-3.5_2.12-1.10.0:
// logs debug
// if EXPIRED and not a metadata table: cache.invalidateAll(metadataTableIdentifiers)
// return ← no close() call
There is no table.io().close() call anywhere in the eviction path.
Proposed fix
In CachingCatalog.java, MetadataTableInvalidatingRemovalListener.onRemoval():
if (value != null && value.io() instanceof Closeable) {
try {
((Closeable) value.io()).close();
} catch (IOException e) {
LOG.warn("Failed to close FileIO for evicted table {}", key, e);
}
}
Note: S3FileIO implements Closeable and its close() method calls S3Client.close(), which shuts down the ScheduledExecutorService and releases all threads. This fix is sufficient to resolve the leak.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start in CachingCatalog.java at MetadataTableInvalidatingRemovalListener.onRemoval(), then inspect S3FileIO.close() to understand the resource lifecycle. Verify that TTL and size eviction close a Closeable FileIO, handle close failures, and release the SDK executor threads without disrupting metadata-table invalidation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100