apache / apache/incubator-xtable
Issue with Metadata Reconciliation Between Iceberg and Delta Tables During Snapshot Updates
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.
### Please describe the bug 🐞
Description
I am encountering an issue while using xtable to perform updates from Iceberg to Delta tables. Here is the observed behavior:
Snapshot 0: The metadata between the Iceberg and Delta tables reconcile as expected.
Snapshot 1: Erroneous metadata is generated that includes "add" and "remove" actions that did not actually occur. This results in a lowered row count in the Delta metadata compared to the source Iceberg table.
Snapshot 2: The metadata appears to reconcile again and reflects the updated values accurately.
Snapshot 3: The issue is recreated with similar discrepancies in the metadata.
Additional Context:
This behavior has been observed consistently to occur in 5 instances across a sample of 30 tables. The issue is in the largest of these tables with around 7 million files and 7.3 trillion records. This table object is "append-only", the disappeared or removed files on snapshot 1 are re-added in snapshot 2. The issue seems cyclical, occurring every alternate snapshot. The only error/info found in the logs is: "incremental sync is not safe from instant falling back to snapshot sync" and "truncated the string representation of a plan since it was too large"
Steps to Reproduce:
Use xtable to perform updates from Iceberg to Delta tables.
Observe metadata reconciliation across snapshots.
Expected Behavior:
The metadata between the Iceberg and Delta tables should reconcile accurately across all snapshots, without erroneous "add" or "remove" actions.
Actual Behavior:
Alternate snapshots (e.g., snapshots 1 and 3) generate erroneous metadata with inaccurate "add" and "remove" actions, leading to a mismatch in row counts.
Environment
Tool: xtable
Source: Apache Iceberg
Destination: Delta Lake
Additional Notes
The issue might be related to how snapshots are processed or metadata is generated.
### Are you willing to submit PR?
- [ ] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Iceberg-to-Delta update across the reported snapshots and inspect the incremental-sync and plan-truncation logs. Done means the generated add/remove metadata and row counts reconcile with the source Iceberg table on every snapshot, including the large append-only table.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100