Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka
Parallel file writing
- 主要语言
- Java
- 星标
- 58
- 派生
- 39
- 平均合并
- 2 天 10 小时
- 30 天内合并 PR
- 5
描述
It looks from the code and observations that the file writing is serial
If there are lots of files to be written then this seems to be limited by latency issues
it seems to me that this could be easily changed
the record writing is currently done like this (in the GCS sink)
```
recordGrouper.records().forEach(this::flushFile);
```
and it could be changed like this
```
recordGrouper
.records()
.entrySet()
.parallelStream()
.forEach(entry -> flushFile(entry.getKey(), entry.getValue()));
```
probably we want some more controls, limiting the grouping, making it optional etc
What are the thoughts of the team on this
贡献指南
评估
这个 Issue 还没有评估数据。