apache / apache/airflow

AF3 TriggerDagRunOperator / DagRunTriggerException should support run_after

Open
#60,443 2 comments 0 reactions 0 assignees View on GitHub
area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

The `TriggerDagRunOperator` raises a `DagRunTriggerException` exception on execute call. The `execute` method also forces a logical date like so:
```python
if self.logical_date is NOTSET:
# If no logical_date is provided we will set utcnow()
parsed_logical_date = timezone.utcnow()
```
Neither support a `run_after` parameter.

### Use case/motivation

To our use cases, having `run_after` and not being limited to `logical_date` was one of the greatest new things in Airflow 3. This is because of the unique index in the database that forces a `dag_id` and `logical_date` to be unique together. Several parallel runs of the same DAG at the same time but with a different `conf` were not possible in Airflow 2. `run_after` effectively enables this as if set, it allows `logical_date` to be `None`.

I think especially an operator that is used to spawn new dag runs during runtime should support this functionality.

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with TriggerDagRunOperator.execute and DagRunTriggerException, tracing how logical_date is currently selected and passed when a DAG run is triggered. Check the related operator and exception tests, then verify that run_after can permit a None logical_date and that triggering still behaves correctly for existing logical-date use cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.