CommitFailedException: Requirement failed: branch main has changed (concurrent writers)
- Dominant language
- Go
- Stars
- 463
- Forks
- 232
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 121
Description
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
Version v0.5.1-0.20260513221951-dafadaa60719
I am trying to write some data into s3 table bucket (newly created empty table) using a deliberately small batch size (1000). I have 8 concurrent writers, each gets a copy of Table object like so:
```
iceTab := check(iceCat.LoadTable(context.Background(), table.Identifier {
iceNamespace,
iceTable,
}))
//...
var wks [workerCount]*worker
//...
for pos := range wks {
wks[pos] = &worker {
id: uint(pos),
iceTab: iceTab,
//...
}
wks[pos].start()
}
//..........
```
Then, each worker executes its own goroutine, where it updates the `iceTab` as following:
```
func (wk *worker) sendBatch(batch arrow.RecordBatch) {
defer batch.Release()
rd := check(array.NewRecordReader(wk.app.schema, []arrow.RecordBatch {
batch,
}))
defer rd.Release()
wk.iceTab = check(wk.iceTab.Append(context.Background(), rd, nil))
}
```
The system works for some time - I've got 29000 rows committed before hitting a cryptic error message:
> CommitFailedException: Requirement failed: branch main has changed: expected id 8388957797659198420 != 3921158795900738659
Would not it be nice to have some sort of trace logging added to the iceberg-go library in order to receive some additional diagnostics?
Contributor guide
Assessment
This issue has not been assessed yet.