apache / apache/iceberg-go

CommitFailedException: Requirement failed: branch main has changed (concurrent writers)

Open
#1,079 6 comments 0 reactions 1 assignee Claimed by @oakad View on GitHub
Dominant language
Go
Stars
463
Forks
232
Avg merge
2d 11h
Merged PRs (30d)
121

Description

### Apache Iceberg version

main (development)

### Please describe the bug 🐞

Version v0.5.1-0.20260513221951-dafadaa60719

I am trying to write some data into s3 table bucket (newly created empty table) using a deliberately small batch size (1000). I have 8 concurrent writers, each gets a copy of Table object like so:

```
iceTab := check(iceCat.LoadTable(context.Background(), table.Identifier {
iceNamespace,
iceTable,
}))
//...
var wks [workerCount]*worker
//...
for pos := range wks {
wks[pos] = &worker {
id: uint(pos),
iceTab: iceTab,
//...
}
wks[pos].start()
}
//..........
```

Then, each worker executes its own goroutine, where it updates the `iceTab` as following:
```
func (wk *worker) sendBatch(batch arrow.RecordBatch) {
defer batch.Release()

rd := check(array.NewRecordReader(wk.app.schema, []arrow.RecordBatch {
batch,
}))

defer rd.Release()

wk.iceTab = check(wk.iceTab.Append(context.Background(), rd, nil))
}
```

The system works for some time - I've got 29000 rows committed before hitting a cryptic error message:
> CommitFailedException: Requirement failed: branch main has changed: expected id 8388957797659198420 != 3921158795900738659

Would not it be nice to have some sort of trace logging added to the iceberg-go library in order to receive some additional diagnostics?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.