Aiven-Open / Aiven-Open/tiered-storage-for-apache-kafka
Too many byte copies during S3 upload
- Ngôn ngữ chính
- Java
- Star
- 239
- Fork
- 59
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
Currently there are three extra bytes copy phase when we upload log segments onto S3:
1. A byte copy from local file to a byte[] in S3ClientWrapper.uploadLogFile: https://github.com/aiven/tiered-storage-for-apache-kafka/blob/main/s3/src/main/java/io/aiven/kafka/tiered/storage/s3/S3ClientWrapper.java#L179
2. A compress and encryption phase in cryptoIOProvider.compressAndEncrypt
3. Another ByteBuffer copy in S3OutputSteram in https://github.com/aiven/tiered-storage-for-apache-kafka/blob/e0c5da2285359c496d2be0c3de3259b2320b9b09/s3/src/main/java/io/aiven/kafka/tiered/storage/s3/S3OutputStream.java#L81
Aiven is hosting for other companies and so it needs to do the data encryption. But in our use case, our data and S3 bucket are only accessible to our company, there is no need to do so many copying and encryption. Can we have an option to skip all those steps?
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.