A NULL column silently suppresses the deadline, and the warning blames a missing DagRun
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.0+
### What happened and how to reproduce it?
A Dag carrying a `DeadlineReference.DAGRUN_LOGICAL_DATE` deadline does not get a `Deadline` row when the DagRun has a NULL `logical_date`. There is no error and no exception, and the only output is a warning naming a DagRun that demonstrably exists:
```
Could not find DagRun for dag_id=..., run_id=...
```
The cause is in `_fetch_from_db` (`serialization/definitions/deadline.py:361`):
```python
result = session.execute(
select(column).where(DagRun.dag_id == dag_id, DagRun.run_id == run_id)
).scalar()
if result is None:
logger.warning("Could not find DagRun for dag_id=%s, run_id=%s", dag_id, run_id)
return result
```
`.scalar()` returns `None` in two different situations: no row matched OR a row matched and the selected column was NULL. The warning assumes the first. For a NULL logical_date it is always the second, and on the DagRun-creation path the row has just been inserted and always exists, so the stated cause is never the real one there. Either way the caller receives `None` and skips the deadline silently.
Two ways to produce a NULL `logical_date`:
1. CLI: `airflow dags trigger `
When you create a Dag this way, unless you set the `-l` flag, the run is not assigned a `logical_date`
2. Asset-triggered: Dag A produces an asset, Dag B consumes it and carries a `DAGRUN_LOGICAL_DATE` deadline.
As above, when Dag B gets created, it gets a `queued_at` timestamp but does not get a `logical_date`.
As a control, run the same Dag using `airflow dags trigger -l "$(date -Iseconds)"` which creates the run with the current time as the `logical_date`.
### What you think should happen instead?
The misleading warning comes from` _fetch_from_db` (`serialization/definitions/deadline.py:361`) which is being resolved in #70714, but the silent skip lives in the caller in `serialization/definitions/dag.py`:
```python
if deadline_time is not None:
session.add(Deadline(...))
# no else
```
Add the `else:` to log which reference was evaluated and which attribute it needed, so a missing `logical_date` is diagnosable rather than invisible. That line is untouched by #70714, so the fix holds both before and after that PR lands.
Separately, while `_fetch_from_db` still exists, it should distinguish "no such DagRun" from "the column is NULL". The row lookup and the column read are different questions, and only the former justifies the current warning. This half becomes moot when #70714 removes the helper, but should likely be done as a seperate PR and backported.
Optionally, a static check: a Dag with `schedule=None` can only be triggered manually or by asset, so a `DAGRUN_LOGICAL_DATE` deadline on one can essentially never fire unless `-l` is passed on every trigger. That is detectable at serialization time and we can alert the user that their deadline can never work.
### Anything else?
This is not specific to logical_date. The same helper serves `DagRunQueuedAtDeadline` (`serialization/definitions/deadline.py:188`), so a run created directly in RUNNING has a NULL `queued_at` and takes the identical path with the identical wrong warning. `get_or_create_dagrun` at `models/dagrun.py:2568` does exactly that. It calls `dag.create_dagrun(..., state=DagRunState.RUNNING)` so deadline processing runs and the queued-at lookup returns `None`. Any nullable column reached through this helper fails the same way.
For completeness: `cli/commands/task_command.py:144` also creates a run in RUNNING, but constructs `DagRun(...)` directly rather than going through `create_dagrun`, so it never reaches deadline processing at all; no deadline row means no warning.
Interaction with #70714: That PR removes `_fetch_from_db` and replaces the lookups with `return dagrun.logical_date` / `return dagrun.queued_at`, so the misleading warning disappears. But the silent skip is in the caller, `if deadline_time is not None:` with no `else` and that part untouched, so after that PR the failure becomes completely silent, with no log line at all, which is worse for diagnosis than today. That PR does add `if new_deadline_time is None: continue` in its recalculation rewrite, so the create path is the remaining gap.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in serialization/definitions/dag.py at the deadline creation conditional and inspect serialization/definitions/deadline.py:_fetch_from_db, including the DagRunQueuedAtDeadline path. Reproduce with a manually triggered DAG lacking -l, or with a NULL queued_at, and trace get_or_create_dagrun in models/dagrun.py. Done means missing deadline attributes are diagnosable and the lookup does not misidentify an existing DagRun as absent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100