Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka

Parallel file writing

未关闭
#311 4 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Java
星标
58
派生
39
平均合并
2 天 10 小时
30 天内合并 PR
5

描述

It looks from the code and observations that the file writing is serial
If there are lots of files to be written then this seems to be limited by latency issues

it seems to me that this could be easily changed

the record writing is currently done like this (in the GCS sink)
```
recordGrouper.records().forEach(this::flushFile);
```
and it could be changed like this
```
recordGrouper
.records()
.entrySet()
.parallelStream()
.forEach(entry -> flushFile(entry.getKey(), entry.getValue()));
```

probably we want some more controls, limiting the grouping, making it optional etc

What are the thoughts of the team on this

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。