Table notification CREATE/UPDATE can fail under concurrent modification without retry
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 137
Description
When Polaris receives a CREATE or UPDATE notification for an external table, the operation can fail if another process modifies the same table at the same time.
Currently:
* UPDATE fails with a concurrent modification error.
* CREATE fails if the table was created by another process before the request completes.
* In both cases, the notification fails instead of retrying.
Simply retrying the same write is not enough because the table information is read only once, making it stale after a concurrent update. A proper fix should:
1. Re-read the latest table entity on every retry.
2. Apply the new metadata location and timestamp again.
3. Retry the update with a limited number of attempts and backoff.
To keep things simple, the initial change could focus only on retrying UPDATE failures caused by concurrent modifications, since that is the most common scenario. FileIO and metadata validation would remain outside the retry loop and run only once.
Question:
Should the CREATE -> UPDATE race condition (where CREATE fails because the entity already exists) be handled in the same change, or should that be a separate follow-up? Also, is there an existing retry utility or pattern in Polaris that should be reused?
Location:
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java
(sendNotificationForTableLike)
Contributor guide
Research direction
Start by reading runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java, especially sendNotificationForTableLike. Trace how CREATE and UPDATE notifications read and write table metadata, then look for an existing retry utility or pattern in Polaris. Done means UPDATE concurrent-modification failures re-read metadata, reapply location and timestamp, and retry with bounded backoff; CREATE handling remains to be decided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100