Aiven-Open / Aiven-Open/tiered-storage-for-apache-kafka
How to increase s3 upload speed
Abierto
- Lenguaje dominante
- Java
- Estrellas
- 239
- Forks
- 59
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
i need upload 200MB/s， but with aws m6in.8xlarge machine and disk io2, it only can reach 120~140MB/s。 so ,i changed the code of S3MultiPartOutputStream.java
Unfortunately, the log uploaded to S3 was parsed using the kafka command and an error was reported
`bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/00000000000000000000.log | head
Dumping /tmp/00000000000000000000.log
Log starting offset: 0
Found 1072768950 invalid bytes at the end of 00000000000000000000.log`
[S3MultiPartOutputStream.java.txt](https://github.com/user-attachments/files/18796731/S3MultiPartOutputStream.java.txt)
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.