apache / apache/airflow

Triggers of a task instance that belongs to a deactivated / paused dag is running

Open
#54,927 6 comments 0 reactions 0 assignees View on GitHub
_eol_affected_version:2.11 area:core kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

2.11.0

### If "Other Airflow 2 version" selected, which one?

N/A

### What happened?

Triggers of task instances that belongs to a deactivated dag continues to run until it fires.

The behaviour is described in the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/2.11.0/authoring-and-scheduling/deferring.html) as such though with little clarification around certain pitfalls like what happens when a dag is deactivated or when paused.

> The trigger runs until it fires, at which point its source task is re-scheduled by the scheduler.

### What you think should happen instead?

When a dag is paused, the triggers tied to the task instance belonging to the dag should be "paused" as well. The triggerer should skip or ignore them. Once the dag is unpaused, the trigger should be ran (and potentially fire). Currently, regardless of the dag (or dagrun) state, the task instance will be marked as scheduled if the trigger fires.

Another variant of the behaviour is when the dag is deactivated which essentially means the dag is "paused". Though in this case, I think the trigger should be deleted and the task marked as failed.

In either case, perhaps, having the trigger fire an event that indicate the corresponding dag has been deactivated or paused is ideal so that trigger can decided what to do in those scenarios.

### How to reproduce

To reproducible in Airflow 2.11.0, I followed these steps:
1. Create a dag with a deferrable TI and start a dagrun
2. Wait for the TI to enter the deferred state
3. Pause the dag
4. Check the task logs

The same behaviour can be reproduced for deactivated dags
1. Create a dag with a deferrable TI and start a dagrun
2. Wait for the TI to enter the deferred state
3. Remove the dag from the dags folder
4. Confirm dag is deactivate and the TI is deferred
5. Check the task logs

I used the `DateTimeSensorAsync` (`apache-airflow-providers-standard==1.6.0`) sensor to demostrate the behaviour.

```python
from airflow import DAG
from airflow.providers.standard.sensors.date_time import DateTimeSensorAsync

from datetime import datetime

with DAG(
dag_id="testing_dag",
schedule_interval="@daily",
start_date=datetime(2023, 1, 1),
catchup=False
) as dag:
c = DateTimeSensorAsync(
task_id='wait_for_time',
target_time="""{{ macros.datetime.utcnow() + macros.timedelta(minutes=1) }}"""
)
```

### Operating System

Debian

### Versions of Apache Airflow Providers

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Reproduce the deferred-task behavior with DateTimeSensorAsync using the steps in the issue, then trace how the triggerer and scheduler handle a paused or deactivated DAG. Compare the observed state transitions with the deferring documentation. Done means the behavior for paused and deactivated DAGs is defined and covered by appropriate verification.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.