apache / apache/arrow-rs-object-store
Object Store HTTP Rate Limiting
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'd like to limit the rate of requests I make to S3, to avoid running into throttling on their side.
**Describe the solution you'd like**
There could be a wrapper, similar to `ThrottledStore`, that puts requests in a queue and executes them up to specified throughput.
**Describe alternatives you've considered**
Rate limiting could be baked into ObjectStore implementations that need it, with rules and configuration that apply to them only. This might be better if there is a wide variety of rate limiting rules between different vendors. On the other hand, this is worse if, say, S3-compatible vendors have different rules between them.
Contributor guide
Research direction
The issue names no files or tests; start by reading the existing ThrottledStore implementation and the ObjectStore request APIs. Determine whether a configurable queue can provide the requested throughput for S3 requests, and define tests that demonstrate the rate limit and completion behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100