apache / apache/airflow

"Queue up new tasks" equivalent in CLI / REST API

Open
#38,342 2 comments 4 reactions 0 assignees View on GitHub
area:API area:CLI kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Description

Suggestion to add a new command in the CLI / REST API to reproduce the "Queue up new tasks" action that can be trigger within the web UI, with `start_date` and `end_date` parameters to allow running this action accross all dag runs lying a given period.

For the CLI, it could be `airflow dags queue_up_new_tasks` or `airflow dags reload` for instance.

### Use case/motivation

Right now, when we add new tasks to an existing dag containing existing dag runs, it is not straightforward to run these newly added tasks in the existing/previous dag runs.

Two workarounds :
- `airflow dags backfill` but not satisfying as it transforms "scheduled" existing dags run into "backfill" dag runs, which prevent us from interacting with them through the web UI (like clearing tasks, etc.)
- `airflow tasks render` followed by `airflow tasks clear` but it is not convenient as the `render` command is only working for one specific execution_date (or run_id) at a time.

Or maybe there are already solutions par that ?

### Related issues

https://github.com/apache/airflow/discussions/30435#discussioncomment-8856597

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by comparing the web UI “Queue up new tasks” action with the existing `airflow dags backfill` and `airflow tasks render`/`clear` commands. Trace the CLI and REST API entry points involved, then clarify the date-range and existing-DAG-run behavior; done means the action is available through the requested interfaces without converting scheduled runs to backfills.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend-api-design, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.