apache / apache/incubator-xtable

Handle log truncation in Delta to Iceberg incremental sync

Open
#779 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.

### Please describe the bug 🐞

There appears to be a discrepancy between how incremental‑sync safety is validated for Delta vs. Iceberg when historical metadata has been truncated. Iceberg correctly detects expired snapshots and triggers a full snapshot sync, but Delta’s `isIncrementalSyncSafeFrom` only checks whether a commit existed at the given instant—not whether the underlying log files still exist. As a result, incremental sync is incorrectly marked “safe,” leading to failures such as `DeltaFileNotFoundException` when attempting to reconstruct old commits. This issue proposes enhancing Delta’s safety check to verify that required commit files are still available before approving incremental sync.

### Are you willing to submit PR?

- [x] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with Delta's isIncrementalSyncSafeFrom check and compare it with Iceberg's handling of expired snapshots. Verify how required historical commit files are checked, then confirm that missing log files trigger a full snapshot sync instead of a DeltaFileNotFoundException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.