google / google/tensorstore

Further S3 Support Umbrella Issue

Open
#114 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.5k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

This issue tracks remaining issues from the TODO list here: https://github.com/google/tensorstore/pull/91#issue-1649705676

Related:

- https://github.com/google/tensorstore/issues/46
- https://github.com/google/tensorstore/pull/91

# KVStore

- [ ] Versioned Bucket Support. From a high-level perspective might involve combining the `x-amz-version-id` and `etag` headers in `StorageGenerations.`
- [ ] S3 specific rate-limiting (per endpoint?). Current S3 rate limiting is based on the GCS rate limiting code. Relevant documentation
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html
- https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
- https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html

# Testing

- [x] ~~Upgrade to localstack 2.3 (release targeted for [26th Sep 2023](https://github.com/localstack/localstack/milestone/4))~~
- ~~`tensorstore::internal::TestKeyValueReadWriteOps` currently succeeds locally on `localstack/localstack:latest` image (c.f. https://github.com/localstack/localstack/issues/9076#issuecomment-1711974643)~~
- [ ] ~~Install localstack in github actions~~
- [x] Moto as S3 testharness. https://github.com/google/tensorstore/commit/8a7f109b4b60b7371929a2c7467516e10671d8f3, https://github.com/google/tensorstore/commit/81aef761d36cff582169fe5b2ab452bcccb6079a
- [ ] Benchmarking See https://github.com/google/tensorstore/pull/91#discussion_r1248172227

# Authorisation

- [x] #119
- [x] #115.
- https://github.com/google/tensorstore/commit/b9b90a21e8c8f7aab92f7699746d70f58890e01f

# Logging

- Review use of [fancy logging](https://github.com/google/tensorstore/commit/00b4519ac8ebcc6f46e2bfaa4f4a3edc2b8faee1) within S3 in general

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.