apache / apache/iceberg

Parallelize writer flush in `DynamicWriter.prepareCommit`

Open
#16,014 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

### Feature description

`DynamicWriter.prepareCommit()` currently flushes each active `TaskWriter` sequentially on the Flink task thread:

https://github.com/apache/iceberg/blob/main/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicWriter.java#L174-L201

```java
for (Map.Entry> entry : writers.entrySet()) {
long startNano = System.nanoTime();
WriteResult writeResult = entry.getValue().complete();
...
}
```

When a subtask is holding many open writers, which is the common case for `DynamicIcebergSink`, since each unique `(table, branch, schemaId, specId, upsertMode, equalityFields)` gets its own `TaskWriter`, checkpoint latency becomes dominated by the **sum** of all per-writer flush times, and the task thread is blocked on IO throughout. Each `TaskWriter.complete()` call ultimately does file-system work (close files, write footers/manifests, stage deletes), so the individual waits are mostly independent and I/O-bound, which means they parallelize well.

### Query engine

Flink

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.