Snowflake: Avoid unbounded FileIO accumulation in SnowflakeCatalog
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
**Apache Iceberg version**
main @ 3038fde68
**Query engine**
N/A (catalog-level, engine-agnostic)
**Please describe the bug**
`SnowflakeCatalog.newTableOps(TableIdentifier)` creates a new `FileIO` on every table load and unconditionally registers it with `closeableGroup` (`snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java` line 255-256, before this fix). `CloseableGroup` backs its closeable list with a `Deque` that only grows until the whole group is closed, so a long-lived `SnowflakeCatalog` instance that repeatedly loads or refreshes tables (a REST catalog server, a long-running Spark session) accumulates one `FileIO` per load, holding native resources (S3/Azure/GCS clients, connection pools) until the catalog itself is closed.
`GlueCatalog.newTableOps(TableIdentifier)` (`aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java`) has the same "new FileIO per table load" design but already solves this with core's `FileIOTracker` (`core/src/main/java/org/apache/iceberg/io/FileIOTracker.java`), which tracks each `FileIO` against its `TableOperations` with a weak-keyed cache and closes it once the `TableOperations` is no longer reachable.
**Steps to reproduce**
1. Create one `SnowflakeCatalog` instance.
2. Call `loadTable()` (or `newTableOps()`) repeatedly, including reloading the same table.
3. Expected: unreferenced `FileIO` instances get reclaimed independently of catalog lifetime, matching `GlueCatalog`. Actual: every call adds a `FileIO` to `closeableGroup` that is never released until `catalog.close()`.
**Additional context**
N/A, covered above.
Contributor guide
Research direction
Start with snowflake/src/main/java/org/apache/iceberg/snowflake/SnowflakeCatalog.java, especially newTableOps(TableIdentifier), and compare it with aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java. Read core/src/main/java/org/apache/iceberg/io/FileIOTracker.java to understand the existing tracking behavior. Done means repeated table loads no longer retain each FileIO until SnowflakeCatalog closes, matching the GlueCatalog approach.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100