apache / apache/iceberg

Snowflake: Avoid unbounded FileIO accumulation in SnowflakeCatalog

Open
#17,095 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

**Apache Iceberg version**
main @ 3038fde68

**Query engine**
N/A (catalog-level, engine-agnostic)

**Please describe the bug**
`SnowflakeCatalog.newTableOps(TableIdentifier)` creates a new `FileIO` on every table load and unconditionally registers it with `closeableGroup` (`snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java` line 255-256, before this fix). `CloseableGroup` backs its closeable list with a `Deque` that only grows until the whole group is closed, so a long-lived `SnowflakeCatalog` instance that repeatedly loads or refreshes tables (a REST catalog server, a long-running Spark session) accumulates one `FileIO` per load, holding native resources (S3/Azure/GCS clients, connection pools) until the catalog itself is closed.

`GlueCatalog.newTableOps(TableIdentifier)` (`aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java`) has the same "new FileIO per table load" design but already solves this with core's `FileIOTracker` (`core/src/main/java/org/apache/iceberg/io/FileIOTracker.java`), which tracks each `FileIO` against its `TableOperations` with a weak-keyed cache and closes it once the `TableOperations` is no longer reachable.

**Steps to reproduce**
1. Create one `SnowflakeCatalog` instance.
2. Call `loadTable()` (or `newTableOps()`) repeatedly, including reloading the same table.
3. Expected: unreferenced `FileIO` instances get reclaimed independently of catalog lifetime, matching `GlueCatalog`. Actual: every call adds a `FileIO` to `closeableGroup` that is never released until `catalog.close()`.

**Additional context**
N/A, covered above.

Contributor guide

Open the contributing guide

Research direction

Start with snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java, especially newTableOps(TableIdentifier), and compare it with aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java. Read core/src/main/java/org/apache/iceberg/io/FileIOTracker.java to understand the existing tracking behavior. Done means repeated table loads no longer retain each FileIO until SnowflakeCatalog closes, matching the GlueCatalog approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.