apache / apache/incubator-xtable

Issue with Metadata Reconciliation Between Iceberg and Delta Tables During Snapshot Updates

Open
#586 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.

### Please describe the bug 🐞

Description
I am encountering an issue while using xtable to perform updates from Iceberg to Delta tables. Here is the observed behavior:

Snapshot 0: The metadata between the Iceberg and Delta tables reconcile as expected.
Snapshot 1: Erroneous metadata is generated that includes "add" and "remove" actions that did not actually occur. This results in a lowered row count in the Delta metadata compared to the source Iceberg table.
Snapshot 2: The metadata appears to reconcile again and reflects the updated values accurately.
Snapshot 3: The issue is recreated with similar discrepancies in the metadata.
Additional Context:
This behavior has been observed consistently to occur in 5 instances across a sample of 30 tables. The issue is in the largest of these tables with around 7 million files and 7.3 trillion records. This table object is "append-only", the disappeared or removed files on snapshot 1 are re-added in snapshot 2. The issue seems cyclical, occurring every alternate snapshot. The only error/info found in the logs is: "incremental sync is not safe from instant falling back to snapshot sync" and "truncated the string representation of a plan since it was too large"
Steps to Reproduce:
Use xtable to perform updates from Iceberg to Delta tables.
Observe metadata reconciliation across snapshots.
Expected Behavior:
The metadata between the Iceberg and Delta tables should reconcile accurately across all snapshots, without erroneous "add" or "remove" actions.

Actual Behavior:
Alternate snapshots (e.g., snapshots 1 and 3) generate erroneous metadata with inaccurate "add" and "remove" actions, leading to a mismatch in row counts.

Environment
Tool: xtable
Source: Apache Iceberg
Destination: Delta Lake
Additional Notes
The issue might be related to how snapshots are processed or metadata is generated.

### Are you willing to submit PR?

- [ ] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Iceberg-to-Delta update across the reported snapshots and inspect the incremental-sync and plan-truncation logs. Done means the generated add/remove metadata and row counts reconcile with the source Iceberg table on every snapshot, including the large append-only table.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.