Custom deadline references registered as TYPES.DAGRUN_QUEUED are not re-anchored when a DagRun is cleared
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.0+
### What happened and how to reproduce it?
The evaluation-timing category passed to `@deadline_reference` is never persisted, and the one place that needs it doesn't actually use it, relying on identifying queued-anchored deadlines by hardcoded class name instead. Custom references are therefore silently excluded.
Clearing a DagRun re-queues it and refreshes `queued_at` (`DagRun.set_state`), and `_recalculate_dagrun_queued_at_deadlines` in `models/taskinstance.py` re-anchors the affected deadlines. Its selection criterion is a string comparison (line 244):
```python
DeadlineAlertModel.reference[ReferenceModels.REFERENCE_TYPE_FIELD].as_string()
== ReferenceModels.DagRunQueuedAtDeadline.__name__,
```
But `reference_type` holds the reference's own bare `__name__`, because
`BaseDeadlineReference.reference_name` returns `self.__class__.__name__` and the base `serialize_reference` emits `{REFERENCE_TYPE_FIELD: self.reference_name}`. Confirmed against stored rows:
```
reference_type | class_path
--------------------------+---------------------------------------
DagRunQueuedAtDeadline | <- built-in, matches
DagRunLogicalDateDeadline |
CloseOfBusinessDeadline | cob_reference.CloseOfBusinessDeadline <- custom, never matches
```
So a custom reference decorated `@deadline_reference(DeadlineReference.TYPES.DAGRUN_QUEUED)` can never satisfy that filter, no matter what it was registered as. `register_custom_reference` appends the class to the in-memory `TYPES.DAGRUN_QUEUED` tuple in whichever process ran the Dag file, and nothing writes that category to `DeadlineAlertModel`, so **the row the query reads has no timing field to filter on even in principle.**
To reproduce:
1. Register a custom reference with `@deadline_reference(DeadlineReference.TYPES.DAGRUN_QUEUED)` whose `_evaluate_with` returns `queued_at` plus something.
2. Put it on one Dag and `DeadlineReference.DAGRUN_QUEUED_AT` on another.
3. Trigger both, record `deadline.deadline_time`.
4. Clear both runs, and compare. The built-in is re-anchored to the new `queued_at`; the custom one is not.
### What you think should happen instead?
The registered timing category should determine which deadlines are re-anchored, so that a custom queued-anchored reference behaves like the built-in one.
1. The `DeadlineReference.TYPES` value needs to be added to the serialized reference, and
2. the check in `_recalculate_dagrun_queued_at_deadlines` which currently says
```python
DeadlineAlertModel.reference[ReferenceModels.REFERENCE_TYPE_FIELD].as_string()
== ReferenceModels.DagRunQueuedAtDeadline.__name__,
```
needs to check that value instead of the `__name__`.
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with _recalculate_dagrun_queued_at_deadlines in models/taskinstance.py, then trace BaseDeadlineReference.serialize_reference, reference_name, register_custom_reference, and the DeadlineAlertModel reference fields. Reproduce the built-in and custom queued-anchored deadlines, clear both DagRuns, and compare their deadline times. Done means the serialized timing category identifies custom queued references and clearing re-anchors them to the refreshed queued_at.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100