Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka
Parallel file writing
- Dominant language
- Java
- Stars
- 58
- Forks
- 39
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 5
Description
It looks from the code and observations that the file writing is serial
If there are lots of files to be written then this seems to be limited by latency issues
it seems to me that this could be easily changed
the record writing is currently done like this (in the GCS sink)
```
recordGrouper.records().forEach(this::flushFile);
```
and it could be changed like this
```
recordGrouper
.records()
.entrySet()
.parallelStream()
.forEach(entry -> flushFile(entry.getKey(), entry.getValue()));
```
probably we want some more controls, limiting the grouping, making it optional etc
What are the thoughts of the team on this
Contributor guide
Assessment
This issue has not been assessed yet.