Dynamic queueing via REST API trigger
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
**Description**
I want to run Airflow DAGs triggered using the REST API in different queue and want to set the queue name on calling the REST API.
**Use case / motivation**
Currently Airflow does support to set queues only on a per DAG deployment level, but in a case where I have a DAG that I want once run with high-prio and 1000s of times with a lower prio, I don't have a chance to make this happen using the REST API (which we need to use for doing programmatic integration with other software / services.
A work around is to deploy the DAG twice with some different hard-coded value for setting the queue, but this will cause huge amount of code and/or code deployment duplications. Also probably causing naming conflicts, etc. Applying work arounds for the part that a feature is missing that I can set the queue for a DAG run using the REST API's data model.
I would want to just send a Json like this to Airflow and it will figure out how to queue the DAG run:
{
"conf": {
... my application config for the DAG run...
}
"queue": "..."
}
**Related Issues**
Not aware of, yet.
Contributor guide
Research direction
Start by reviewing Airflow's REST API DAG-trigger request model and the scheduler's DAG-run queue handling. The issue does not name files or tests; done would mean a REST-triggered run can receive a queue name alongside conf and be scheduled using that requested queue, with coverage for the request behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100