apache / apache/arrow-rs-object-store

Object Store HTTP Rate Limiting

Open
#268 17 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue help wanted
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'd like to limit the rate of requests I make to S3, to avoid running into throttling on their side.

**Describe the solution you'd like**
There could be a wrapper, similar to `ThrottledStore`, that puts requests in a queue and executes them up to specified throughput.

**Describe alternatives you've considered**
Rate limiting could be baked into ObjectStore implementations that need it, with rules and configuration that apply to them only. This might be better if there is a wide variety of rate limiting rules between different vendors. On the other hand, this is worse if, say, S3-compatible vendors have different rules between them.

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests; start by reading the existing ThrottledStore implementation and the ObjectStore request APIs. Determine whether a configurable queue can provide the requested throughput for S3 requests, and define tests that demonstrate the rate limit and completion behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.