apache / apache/airflow

Support updating DagRun note from notifier/callback context in Airflow 3

Open
#64,890 2 comments 0 reactions 0 assignees View on GitHub
area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

In Airflow 2, a common notifier/callback pattern was:

1. Post an alert to an external system such as Slack.
2. Persist a backlink or correlation identifier on the `DagRun` note.
3. Reuse that stored link later to update the original alert if the DagRun is cleared and eventually succeeds.

That migration path is not straightforward in Airflow 3.

In Airflow 3 callback/notifier code, `context["dag_run"]` is no longer the old SQLAlchemy `DagRun` contract that many Airflow 2 implementations relied on, and the Task SDK client does not appear to expose a supported helper for updating `DagRun.note` from callback/notifier runtime code.

There is already public API support for DagRun note patching, so this request is specifically about having a supported callback-safe/runtime-safe way to do that from Airflow 3 notifier code.

### Use case/motivation

A concrete use case is a DAG-level failure notifier that:

1. Fires once when the DagRun fails.
2. Collects failed task IDs for that run.
3. Posts a Slack message.
4. Stores the Slack thread URL in the DagRun note for traceability.
5. On a later callback (for example after the DagRun is cleared and succeeds), reads that stored link and updates the original Slack message instead of sending a brand new one.

The "list failed tasks" part is portable in Airflow 3 via `context["task_instance"].get_task_states(...)` in a real callback execution path.

The missing piece is the note update. In Airflow 2 this was commonly done with ORM/session access such as `DagRunNote`, `dag_run.dag_run_note`, and `provide_session`. In Airflow 3 that pattern is not portable, but I could not find a supported replacement in notifier/callback context.

### Related issues

- #51816 is related only in the sense that local `dag.test()` / `airflow dags test` debugging can be misleading for callback runtime APIs. It does not solve the DagRun note update gap described here.

### Requested outcome

One of the following would unblock Airflow 2 -> 3 notifier migrations cleanly:

1. Add a supported Task SDK / callback-context helper to update `DagRun.note` from notifier code.
2. Expose a documented runtime client operation for patching the current DagRun note from callback code.
3. If this is intentionally unsupported, document the recommended replacement pattern for storing callback correlation metadata on a DagRun in Airflow 3.

### Environment where this came up

- Apache Airflow 3.1.8
- DAG-level `on_failure_callback`
- Local Astro / Astronomer runtime during migration testing

### Minimal callback shape

```python
from airflow.notifications.basenotifier import BaseNotifier

class ExampleNotifier(BaseNotifier):
def notify(self, context):
dag_run = context["dag_run"]
ti = context["task_instance"]

task_states = ti.get_task_states(
dag_id=dag_run.dag_id,
run_ids=[dag_run.run_id],
)

failed_tasks = [
task_id
for task_id, state in task_states.get(dag_run.run_id, {}).items()
if state in {"failed", "upstream_failed"}
]

# Supported in Airflow 3: derive failed task list.
print(failed_tasks)

# Missing piece: supported way to persist a backlink such as a Slack URL
# onto the DagRun note from this notifier/callback context.
```

### Why I think this belongs in OSS Airflow

This appears to be a migration and API-surface gap rather than a managed-service issue:

- Airflow 3 callback context behavior changed relative to many Airflow 2 implementations.
- Public DagRun note patching exists, so the underlying capability is present.
- The missing piece is a supported, documented path for callback/notifier code to update the current DagRun note.

Thanks for taking a look.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.