apache / apache/airflow

Impose rate limits for task starts as pool feature

Open
#15,082 4 comments 4 reactions 0 assignees View on GitHub
area:scheduler kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

**Description**

As a pool feature, queue task starts such that task initiation is distributed over time according to a sliding window rate limit (may be more easily implemented as task initiation cool down within a pool).

**Use case / motivation**

APIs commonly will impose certain techniques to limit rate of requests (sliding window, fixed window, token bucket, leaky bucket, etc). While task retries may resolve the issue, all failures could potentially be avoided if there was a feature to match the endpoint's programatic behavior (I suggest sliding window as it has other benefits).

A sliding window rate limiter could also be used to stagger task/request initiation to a legacy system. There are a number of reasons why you may want to stagger requests to a legacy system, such as if the beginning portion of a request is the most resource intensive within the foreign system, or if the legacy system itself does not itself provide its own rate-limiting signals.

It may be more easily implemented as a cool down between a pooled task initiation and the next queued task start, but figured I'd frame the feature request to match other rate-limiting strategies and techniques commonly seen.

**Related Issues**

Potentially https://github.com/apache/airflow/issues/8789 ?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Airflow's existing pool and scheduler behavior, then compare the related issue #8789 before choosing a sliding-window or cooldown design. Define how queued task starts are limited per pool and how the behavior is tested; done means the selected pool rate limit works without changing unrelated scheduling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.