Prevent unsafe metadata cleanup after ambiguous JDBC table commits
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 140
Description
## Problem
During an Iceberg table metadata commit, Polaris writes the new metadata location to the metastore through an auto-commit JDBC update.
If the database applies the update but the JDBC client loses the response, Polaris cannot determine whether the write committed. The current behavior can report a definite optimistic-concurrency failure on a subsequent attempt.
Iceberg interprets a definite commit failure as permission to clean up files created for the commit. If the original metastore update actually succeeded, those files may already be referenced by the committed metadata. Cleanup can therefore remove manifests or metadata files required to read the table, resulting in table corruption.
## Expected behavior
When a JDBC update fails during `executeUpdate()` and its outcome may be unknown:
1. Do not retry the update blindly.
2. Reload the persisted entity.
3. If its metadata location matches the requested location, treat the commit as successful.
4. If the intended result cannot be proven, return `CommitStateUnknownException` rather than a definite commit conflict.
A commit-state-unknown response causes the client to preserve commit artifacts rather than deleting files that may have been committed.
## Scope
- Applies to Iceberg table-like metadata updates backed by relational JDBC persistence.
- Normal retry behavior remains unchanged for failures known to occur before execution, including connection acquisition and statement preparation.
- Does not depend on or require a specific external connection pooler.
Contributor guide
Research direction
Start by tracing the JDBC executeUpdate() path for Iceberg table-like metadata updates and how failures are classified. Follow the persisted-entity reload and metadata-location comparison behavior; done means pre-execution failures retain normal retries, while ambiguous outcomes are proven successful or returned as CommitStateUnknownException without unsafe cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100