apache / apache/airflow

calculate_dagrun_date_fields doesn't understand how it is called

Open
#59,618 5 comments 0 reactions 0 assignees View on GitHub
area:core area:scheduler kind:bug kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Body

It has an argument `last_automated_run` but in reality, what it is given is not always the latest run.

This is because sometimes it is called, in the scheduler, with whatever run the scheduler happens to be dealing with -- which might not be the latest run.

And even if something _thinks_ it is passing the latest run, due to airflow being a distributed system, something else could create a new run in the meantime, making it not the latest run anymore.

For one thing, this behavior sometimes causes airflow to set the next_dagrun field back in time.

We should probably at least rename this field. But it might also be worth looking at how it's called and possibly making some adjustments to the logic.

### Committer

- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start at calculate_dagrun_date_fields and the scheduler call sites described in the issue. Trace whether last_automated_run is actually the latest run, then define tests covering next_dagrun moving backward; done means the argument meaning and scheduler behavior are consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.