[C++] Mechanism for throttling remote filesystems to avoid rate limiting
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the enhancement requested
In the current implementation of at least the S3 filesystem, it's possible to hit AWS' rate limiting and this makes it hard to use a function like `pyarrow.parquet.write_dataset` to write highly partitioned datasets to S3. When the rate limit is hit during writing, the user receives an HTTP 503 with a "SLOW_DOWN" error and the request needs to be retried. I would guess any other cloud storage provider (ie GCS) would behave similarly though I haven't tested.
There are a few things that could be done to address this issue:
- Add logic to the S3 implementation to deal with the issue directly
- Implement a generic `ThrottledFilesystem` that can wrap another `Filesystem` and throttle certain calls (ie writes)
- Expose the existing `SlowFileSystem` to Python/R and possibly tweak its implementation to suit this use case
### Component(s)
C++
Contributor guide
Research direction
Start by reading the S3 filesystem implementation and the existing SlowFileSystem entry point, then trace how pyarrow.parquet.write_dataset reaches remote filesystem writes. Compare the three proposed approaches and confirm the intended scope with maintainers; the issue does not yet define which design or tests would constitute done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp
- Domain
- cloud, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100