Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka

Parallel file writing

Offen
#311 4 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Java
Sterne
58
Forks
39
Ø Merge
2 T. 10 Std.
Gemergte PRs (30 T.)
5

Beschreibung

It looks from the code and observations that the file writing is serial
If there are lots of files to be written then this seems to be limited by latency issues

it seems to me that this could be easily changed

the record writing is currently done like this (in the GCS sink)
```
recordGrouper.records().forEach(this::flushFile);
```
and it could be changed like this
```
recordGrouper
.records()
.entrySet()
.parallelStream()
.forEach(entry -> flushFile(entry.getKey(), entry.getValue()));
```

probably we want some more controls, limiting the grouping, making it optional etc

What are the thoughts of the team on this

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.