Rewrite data files reports partial commit failure even when all commits succeed
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
### Summary
When running the `rewrite_data_files` action with partial progress enabled (`partial-progress.enabled=true`), the action can report commit failures — emitting a `WARN` log, or even throwing a `RuntimeException` when `partial-progress.max-failed-commits` is exceeded — **even though every commit actually succeeded**.
### Root cause
`RewriteDataFilesSparkAction#doExecuteWithPartialProgress` does not count failed commits directly. Instead it infers the count by subtraction:
```java
int totalCommits = Math.min(plan.totalGroupCount(), maxCommits);
int failedCommits = totalCommits - commitService.succeededCommits();
```
This assumes the total number of commits equals `min(totalGroupCount, maxCommits)`. But the action groups multiple file groups into a single commit. The number of file groups per commit is:
```
groupsPerCommit = ceil(totalGroupCount / maxCommits)
```
so the actual number of commits is:
```
ceil(totalGroupCount / groupsPerCommit)
```
which is usually **smaller** than `min(totalGroupCount, maxCommits)`. The difference is then mis-reported as failed commits.
### Example
- `totalGroupCount = 70`, `maxCommits = 20`
- `groupsPerCommit = ceil(70 / 20) = 4`
- actual commits when all succeed = `ceil(70 / 4) = 18`
- reported `failedCommits = min(70, 20) - 18 = 2`
So 2 phantom failures are reported even though all 18 commits succeeded. If `max-failed-commits < 2`, the whole operation fails with an exception despite a fully successful rewrite.
### Expected behavior
`failedCommits` should reflect the number of commits that actually threw during `commitOrClean`, and should be `0` when every commit succeeds.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start in RewriteDataFilesSparkAction#doExecuteWithPartialProgress and trace how commitOrClean records successful and failed commits. Check the partial-progress handling around commitService.succeededCommits(), then verify that an all-successful rewrite reports zero failed commits and does not exceed max-failed-commits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100