apache / apache/iceberg

CachingCatalog does not close FileIO on cache eviction, causing S3FileIO / SDK v2 thread leak in long-running applications

Open Beginner friendly
#15,898 17 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.10.0

### Query engine

Spark

### Please describe the bug 🐞

Apache Iceberg version: 1.10.0

Component: org.apache.iceberg.CachingCatalog

Description

CachingCatalog uses a Caffeine cache to hold Table objects. When an entry is evicted (by TTL via cache.expiration-interval-ms or by size via cache.max-total-bytes), the RemovalListener
(MetadataTableInvalidatingRemovalListener) only invalidates related metadata table entries. It does not call table.io().close().

This means any resources held by the FileIO implementation are never released on eviction.

Impact

With io-impl = org.apache.iceberg.aws.s3.S3FileIO:
- Each evicted Table leaves behind a live AWS SDK v2 S3Client
- Each S3Client owns a ScheduledExecutorService (sdk-ScheduledExecutor-N) with background threads for credential refresh (IMDSv2)
- These threads are GC roots — they can never be collected
- In a long-running process (e.g. Spark Thrift Server), threads accumulate without bound until the JVM crashes with os::commit_memory failed; error='Not enough space' (errno=12)

Observed in production (Spark Thrift Server, ~24h uptime):
Total JVM threads: 27,877
sdk-ScheduledExecutor: 27,657
Distinct pool instances: 8,075+

Proof from bytecode

CachingCatalog$MetadataTableInvalidatingRemovalListener.onRemoval() decompiled from iceberg-spark-runtime-3.5_2.12-1.10.0:

// logs debug
// if EXPIRED and not a metadata table: cache.invalidateAll(metadataTableIdentifiers)
// return ← no close() call

There is no table.io().close() call anywhere in the eviction path.

Proposed fix

In CachingCatalog.java, MetadataTableInvalidatingRemovalListener.onRemoval():

if (value != null && value.io() instanceof Closeable) {
try {
((Closeable) value.io()).close();
} catch (IOException e) {
LOG.warn("Failed to close FileIO for evicted table {}", key, e);
}
}

Note: S3FileIO implements Closeable and its close() method calls S3Client.close(), which shuts down the ScheduledExecutorService and releases all threads. This fix is sufficient to resolve the leak.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start in CachingCatalog.java at MetadataTableInvalidatingRemovalListener.onRemoval(), then inspect S3FileIO.close() to understand the resource lifecycle. Verify that TTL and size eviction close a Closeable FileIO, handle close failures, and release the SDK executor threads without disrupting metadata-table invalidation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.