apache / apache/arrow

[C++] Mechanism for throttling remote filesystems to avoid rate limiting

Open
#34,892 10 comments 2 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

In the current implementation of at least the S3 filesystem, it's possible to hit AWS' rate limiting and this makes it hard to use a function like `pyarrow.parquet.write_dataset` to write highly partitioned datasets to S3. When the rate limit is hit during writing, the user receives an HTTP 503 with a "SLOW_DOWN" error and the request needs to be retried. I would guess any other cloud storage provider (ie GCS) would behave similarly though I haven't tested.

There are a few things that could be done to address this issue:

- Add logic to the S3 implementation to deal with the issue directly
- Implement a generic `ThrottledFilesystem` that can wrap another `Filesystem` and throttle certain calls (ie writes)
- Expose the existing `SlowFileSystem` to Python/R and possibly tweak its implementation to suit this use case

### Component(s)

C++

Contributor guide

Open the contributing guide

Research direction

Start by reading the S3 filesystem implementation and the existing SlowFileSystem entry point, then trace how pyarrow.parquet.write_dataset reaches remote filesystem writes. Compare the three proposed approaches and confirm the intended scope with maintainers; the issue does not yet define which design or tests would constitute done.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp
Domain
cloud, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.