Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka

Parallel file writing

Open
#311 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
58
Forks
39
Avg merge
2d 10h
Merged PRs (30d)
5

Description

It looks from the code and observations that the file writing is serial
If there are lots of files to be written then this seems to be limited by latency issues

it seems to me that this could be easily changed

the record writing is currently done like this (in the GCS sink)
```
recordGrouper.records().forEach(this::flushFile);
```
and it could be changed like this
```
recordGrouper
.records()
.entrySet()
.parallelStream()
.forEach(entry -> flushFile(entry.getKey(), entry.getValue()));
```

probably we want some more controls, limiting the grouping, making it optional etc

What are the thoughts of the team on this

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.