[Feature] [core] Iceberg: createMetadataWithoutBase should include full history of Paimon snapshots
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Code path: https://github.com/apache/paimon/blob/ff02f6bf3ceccf8dcac38bc58cf6db390509bd46/paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java#L305C49-L305C59
createMetadataWithoutBase is called under certain conditions like
- enabling Iceberg compatibility for the first time
- committing Iceberg metadata after a previous commit failed in the Iceberg layer (for many possible reasons)
In some cases the Paimon table will have many snapshots already (e.g. snapshots 1 to 1000) and after calling createMetadataWithoutBase, only the latest Paimon commit will be synced to Iceberg.
By syncing the whole Paimon history to Iceberg, the Iceberg compatibility feature becomes suitable for production use cases that require Iceberg time travel.
My use case is this:
- Sync MySQL tables to Paimon using Flink CDC
- Tag daily Paimon snapshots [automatically](https://paimon.apache.org/docs/master/maintenance/manage-tags/)
- Iceberg readers read daily snapshots
When a failure happens in the Iceberg committer the metadata needs to be recreated on the next commit. Only the latest Paimon snapshot is included. The daily snapshot component of this pipeline has broken and cannot be recovered.
### Solution
A simple solution may exist. Assume snapshot range [x, y]:
If creating metadata without base:
- createMetadataWithoutBase for the earliest snapshot, x
- (for i=x, i <= y, i++) call createMetadataWithBase(i)
### Anything else?
Consider making this feature opt-in with configuration as it may be a costly operation to sync many Paimon snapshots to Iceberg and therefore reach the Flink checkpoint timeout.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java at createMetadataWithoutBase. Trace how the snapshot range is selected and how createMetadataWithBase is used. Done means recreating Iceberg metadata includes the full relevant Paimon snapshot history, while accounting for the stated checkpoint-timeout concern.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100