Spark's RewriteDataFiles fails if duplicate files exist in a single live manifest
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
## Bug Report
Spark's RewriteDataFiles action fails with `Invalid REPLACE operation: %s added records > %s replaced records` for tables where duplicate files exist in the same manifest. This can occur when a file is appended more than once in separate APPENDs -> a RewriteManifests maintenance action happens -> a RewriteDataFiles maintenance action happens.
This failure occurs during SnapshotProducer's [apply method](https://github.com/apache/iceberg/blob/8c1ee9d9df4184a7889f85f6a9e1f82f959e0eaf/core/src/main/java/org/apache/iceberg/SnapshotProducer.java#L349-L355), which checks that `added-records` <= `replaced-records`.
My understanding is that, assuming we have 2 snapshots with a duplicate file appearing in each that are later joined into a single live manifest by RewriteManifests:
* `added-records` is calculated by checking how many records exist in all newly-added files in the newly-rewritten snapshot; it contains duplicate rows from the duplicate file. [MergingSnapshotProducer#add](https://github.com/apache/iceberg/blob/8c1ee9d9df4184a7889f85f6a9e1f82f959e0eaf/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L252) -> [SnapshotSummary#addedFile](https://github.com/apache/iceberg/blob/8c1ee9d9df4184a7889f85f6a9e1f82f959e0eaf/core/src/main/java/org/apache/iceberg/SnapshotSummary.java#L118-L121) -> [SnapshotSummary#updatePartitions](https://github.com/apache/iceberg/blob/8c1ee9d9df4184a7889f85f6a9e1f82f959e0eaf/core/src/main/java/org/apache/iceberg/SnapshotSummary.java#L165-L166) -> [SnapshotSummary.UpdateMetrics#addedFile](https://github.com/apache/iceberg/blob/8c1ee9d9df4184a7889f85f6a9e1f82f959e0eaf/core/src/main/java/org/apache/iceberg/SnapshotSummary.java#L296)
* `deleted-records` (replacedRecords in the check) is calculated by checking the existing files *after the duplicate files have been removed from the manifest*; the filtering happens in [SnapshotProducer#apply](https://github.com/apache/iceberg/blob/8c1ee9d9df4184a7889f85f6a9e1f82f959e0eaf/core/src/main/java/org/apache/iceberg/SnapshotProducer.java#L302-L354), which applies the manifest filtering logic before checking the validation
* In fact, [the comments in the same file indicate that the snapshot summary metrics require files to be tracked for them to be accurate](https://github.com/apache/iceberg/blob/8c1ee9d9df4184a7889f85f6a9e1f82f959e0eaf/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L548).
* the ManifestFilterManager filters files that appear twice in the manifests (so removes one copy of the duplicate file from the tracked set of files), and increments the `deleted-duplicate-files`: [ManifestFilterManager#filterManifestWithDeletedFiles](https://github.com/apache/iceberg/blob/8c1ee9d9df4184a7889f85f6a9e1f82f959e0eaf/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java#L540-L545) does not increment the `deleted-records` metric. We *do* see this log line in our compaction/RewriteDataFiles action job logs.
* So later in SnapshotProducer#apply, the replacedRecords undercounts by the number of rows in the duplicate file(s), and our RewriteDataFiles action fails.
There are several reasons why duplicate files can appear, in our case it's because we have to decouple our write-files-to-S3 step from our commit-to-Iceberg step. This possibility is implicitly acknowledged by the fact that the ManifestFilterManager filters duplicates. We shouldn't fail to apply the update due to faulty bookkeeping.
## Repro:
1. Create an Iceberg table with your choice of catalog, object store, etc.
2. Add the same file to the table 2+ times using e.g. a FastAppend
3. Run a RewriteManifests action
4. Run a RewriteDataFiles action -> triggers bug
I have pushed a minimal repro [here](https://github.com/snicholasbarton/iceberg-dupe-repro)
## Suggested Fix:
I think the simplest fix, that just changes the bookkeeping of files and records "deleted" (that is, number of records in deleted files), is to keep track of every removed manifest entry in addition to the unique deleted manifest entries in the snapshot summary via an additional collection in ManifestFilterManager#filterManifestWithDuplicateEntries. This way we can accurately count files and records "deleted" by the file rewrite.
## AI Disclosure:
I used Claude Code (Opus 4.8) to explore the repo and help identify the issue, plus port our internal repro to something open source.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Reproduce the failure with the linked minimal repro by appending the same file twice, running RewriteManifests, then RewriteDataFiles. Read ManifestFilterManager#filterManifestWithDeletedFiles and the duplicate-entry handling, then follow SnapshotProducer#apply and SnapshotSummary metrics. Done means RewriteDataFiles applies successfully and deleted-records bookkeeping accounts for duplicate entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100