apache / apache/arrow-rs-object-store

move s3 express session token fetching out of request path

Open
#649 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

requests to S3 Express require creating and refreshing a 'Session Token' every 5 minutes.
This `CreateSession` API calls takes > 300ms to complete.

Currently `CreateSession` is implictly called by object-store lib whenever a request is made and the token is out of date. This however greatly increases the latency of the request itself.
For example, if you were doing a GET request that you expected to complete in single digit miliseconds, due to the `CreateSession` call being bundled with it, you now get a response time of >300 ms.

**Describe the solution you'd like**
Session Tokens should be refreshed in the background, instead of synchronously as part of request to S3 Express. Current implementation can cause massive p99 increases by bundling the `CreateSession` call with any PUT/GET call that is expected to take single digit ms.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by tracing the object-store library's S3 Express request path and its CreateSession token handling. Determine how token refresh can run in the background without delaying PUT or GET requests, then verify that requests no longer wait for the roughly 300 ms session call when a token expires.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.