aws-samples / aws-samples/sample-s3-hybrid-cache
Cache ListObjectsV2
Open
enhancement
- Dominant language
- Rust
- Stars
- 38
- Forks
- 7
- Avg merge
- 13h 23m
- Merged PRs (30d)
- 1
Description
Optional caching of [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html)
Particularly valuable for
1. Multiple clients requesting the same listing at the same/similar time
2. Very large object counts. Benchmarks against Amazon S3 from https://github.com/aws-samples/sample-s3-listobjectversions-to-csv/ : 1 million = 5 mins, 3M = 15 mins, 60M = 5.5 hours.
3. Buckets with many noncurrent versions (including delete markers) which can impact ListObjectsV2 performance (ref https://repost.aws/knowledge-center/s3-troubleshoot-unresponsive-list)
Contributor guide
Assessment
This issue has not been assessed yet.