Aiven-Open / Aiven-Open/tiered-storage-for-apache-kafka
How to increase s3 upload speed
Open
- Dominant language
- Java
- Stars
- 239
- Forks
- 59
- PR merge metrics
- No merged PRs in 30d
Description
i need upload 200MB/s， but with aws m6in.8xlarge machine and disk io2, it only can reach 120~140MB/s。 so ,i changed the code of S3MultiPartOutputStream.java
Unfortunately, the log uploaded to S3 was parsed using the kafka command and an error was reported
`bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/00000000000000000000.log | head
Dumping /tmp/00000000000000000000.log
Log starting offset: 0
Found 1072768950 invalid bytes at the end of 00000000000000000000.log`
[S3MultiPartOutputStream.java.txt](https://github.com/user-attachments/files/18796731/S3MultiPartOutputStream.java.txt)
Contributor guide
Assessment
This issue has not been assessed yet.