Aiven-Open / Aiven-Open/tiered-storage-for-apache-kafka
Too many byte copies during S3 upload
- Vorherrschende Sprache
- Java
- Sterne
- 239
- Forks
- 59
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
Currently there are three extra bytes copy phase when we upload log segments onto S3:
1. A byte copy from local file to a byte[] in S3ClientWrapper.uploadLogFile: https://github.com/aiven/tiered-storage-for-apache-kafka/blob/main/s3/src/main/java/io/aiven/kafka/tiered/storage/s3/S3ClientWrapper.java#L179
2. A compress and encryption phase in cryptoIOProvider.compressAndEncrypt
3. Another ByteBuffer copy in S3OutputSteram in https://github.com/aiven/tiered-storage-for-apache-kafka/blob/e0c5da2285359c496d2be0c3de3259b2320b9b09/s3/src/main/java/io/aiven/kafka/tiered/storage/s3/S3OutputStream.java#L81
Aiven is hosting for other companies and so it needs to do the data encryption. But in our use case, our data and S3 bucket are only accessible to our company, there is no need to do so many copying and encryption. Can we have an option to skip all those steps?
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.