Remove incorrect snapshotId filtering in RewriteTablePathSparkAction
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.10.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
The snapshotId filtering logic was incorrectly excluding live data files during table copy operations. entry.snapshotId() records when a data file was initially added, not which snapshots currently reference it. After manifest compaction or snapshot expiration, a snapshot can reference manifests containing entries with expired snapshotIds, but those files are still live and must be copied.
The check snapshotIds.contains(entry.snapshotId()) was fundamentally wrong because it filtered out data files whose original snapshot had expired, even though they were still referenced by the snapshot(s) being copied.
This bug likely affects many production tables where manifest compaction has run. Customers may not notice the issue if they don't query the missing data files.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start at RewriteTablePathSparkAction and trace the snapshotIds filtering around entry.snapshotId(), focusing on how manifest compaction and snapshot expiration affect live files. Done means data files still referenced by the snapshots being copied are retained, with coverage for the described case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100