Parallelize writer flush in `DynamicWriter.prepareCommit`
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
### Feature description
`DynamicWriter.prepareCommit()` currently flushes each active `TaskWriter` sequentially on the Flink task thread:
https://github.com/apache/iceberg/blob/main/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicWriter.java#L174-L201
```java
for (Map.Entry> entry : writers.entrySet()) {
long startNano = System.nanoTime();
WriteResult writeResult = entry.getValue().complete();
...
}
```
When a subtask is holding many open writers, which is the common case for `DynamicIcebergSink`, since each unique `(table, branch, schemaId, specId, upsertMode, equalityFields)` gets its own `TaskWriter`, checkpoint latency becomes dominated by the **sum** of all per-writer flush times, and the task thread is blocked on IO throughout. Each `TaskWriter.complete()` call ultimately does file-system work (close files, write footers/manifests, stage deletes), so the individual waits are mostly independent and I/O-bound, which means they parallelize well.
### Query engine
Flink
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Assessment
This issue has not been assessed yet.