Aiven-Open / Aiven-Open/tiered-storage-for-apache-kafka

S3 key/folder structure for performance

Open
#124 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
239
Forks
59
PR merge metrics
No merged PRs in 30d

Description

For S3 file transfer performance, the S3 objects prefix structure is the key for high performance.

Do you know how the folder structure on S3 is laid out? Is that using a simple hierarchical directory path (e.g. [s3://bucket/tiered/topic-1/parttion-1/logsetment-1.log](http://s3//bucket/tiered/topic-1/parttion-1/logsetment-1.log))?

This layout would bottleneck on reading/writing into topic-1/partition-1 since S3 partitioning is on the topic level path (tiered/topic-1), if we can add a random salt hash part into the path (e.g. s3://bucket/tiered/aabbg/topic-1) this would greatly improve the S3 transfer throughput.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.