apache / apache/airflow

Dynamic queueing via REST API trigger

Open
#11,080 6 comments 1 reaction 0 assignees View on GitHub
area:API area:core area:scheduler kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

**Description**

I want to run Airflow DAGs triggered using the REST API in different queue and want to set the queue name on calling the REST API.

**Use case / motivation**

Currently Airflow does support to set queues only on a per DAG deployment level, but in a case where I have a DAG that I want once run with high-prio and 1000s of times with a lower prio, I don't have a chance to make this happen using the REST API (which we need to use for doing programmatic integration with other software / services.

A work around is to deploy the DAG twice with some different hard-coded value for setting the queue, but this will cause huge amount of code and/or code deployment duplications. Also probably causing naming conflicts, etc. Applying work arounds for the part that a feature is missing that I can set the queue for a DAG run using the REST API's data model.

I would want to just send a Json like this to Airflow and it will figure out how to queue the DAG run:

{
"conf": {
... my application config for the DAG run...
}
"queue": "..."
}

**Related Issues**

Not aware of, yet.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Airflow's REST API DAG-trigger request model and the scheduler's DAG-run queue handling. The issue does not name files or tests; done would mean a REST-triggered run can receive a queue name alongside conf and be scheduled using that requested queue, with coverage for the request behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.