Aiven-Open / Aiven-Open/tiered-storage-for-apache-kafka
How to increase s3 upload speed
Offen
- Vorherrschende Sprache
- Java
- Sterne
- 239
- Forks
- 59
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
i need upload 200MB/s， but with aws m6in.8xlarge machine and disk io2, it only can reach 120~140MB/s。 so ,i changed the code of S3MultiPartOutputStream.java
Unfortunately, the log uploaded to S3 was parsed using the kafka command and an error was reported
`bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/00000000000000000000.log | head
Dumping /tmp/00000000000000000000.log
Log starting offset: 0
Found 1072768950 invalid bytes at the end of 00000000000000000000.log`
[S3MultiPartOutputStream.java.txt](https://github.com/user-attachments/files/18796731/S3MultiPartOutputStream.java.txt)
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.