Impose rate limits for task starts as pool feature
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
**Description**
As a pool feature, queue task starts such that task initiation is distributed over time according to a sliding window rate limit (may be more easily implemented as task initiation cool down within a pool).
**Use case / motivation**
APIs commonly will impose certain techniques to limit rate of requests (sliding window, fixed window, token bucket, leaky bucket, etc). While task retries may resolve the issue, all failures could potentially be avoided if there was a feature to match the endpoint's programatic behavior (I suggest sliding window as it has other benefits).
A sliding window rate limiter could also be used to stagger task/request initiation to a legacy system. There are a number of reasons why you may want to stagger requests to a legacy system, such as if the beginning portion of a request is the most resource intensive within the foreign system, or if the legacy system itself does not itself provide its own rate-limiting signals.
It may be more easily implemented as a cool down between a pooled task initiation and the next queued task start, but figured I'd frame the feature request to match other rate-limiting strategies and techniques commonly seen.
**Related Issues**
Potentially https://github.com/apache/airflow/issues/8789 ?
Contributor guide
Research direction
Start by reviewing Airflow's existing pool and scheduler behavior, then compare the related issue #8789 before choosing a sliding-window or cooldown design. Define how queued task starts are limited per pool and how the behavior is tested; done means the selected pool rate limit works without changing unrelated scheduling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100