Manual run with same execution date as scheduled run causes the scheduled run to be skipped.
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
main (development)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When there is a manual dagrun, it is created with the logical date as data interval end. In case of scheduled runs the logical date is the data interval start. In this case when a future run is scheduled there is a check for existing dagruns in the loop with respect to execution date (logical date) which causes the dagrun to be skipped since it exists from the manual run with a different data interval.
This looks like a known issue in 2.x and can to be changed in 3.x . Related issue https://github.com/apache/airflow/issues/41014
https://github.com/apache/airflow/blob/4d849e87ecce654e100c36dca3d0fbe19bcd8b6b/airflow/jobs/scheduler_job_runner.py#L1314-L1348
### What you think should happen instead?
Scheduled dag run should be executed though a manual run with same execution date exists.
### How to reproduce
1. Place the below dag and parse it.
2. Before unpausing the dag trigger a manual run with logical date as "2024-07-01T00:00:00+00".
3. A manual run is triggered with run_id as manual__2024-07-01T00:00:00+00:00 and exeuction_date as 2024-07-01 00:00:00+00:00. Data interval is (2024-06-01, 00:00:00 UTC, 2024-07-01, 00:00:00 UTC)
4. The run with data interval (2024-07-01T00:00:00+00:00, 2024-08-01T00:00:00+00:00) is not scheduled.
```python
from datetime import datetime, timedelta
from airflow import DAG
from airflow.decorators import task
with DAG(
dag_id="execution_date_debug",
start_date=datetime(2024, 1, 1),
catchup=True,
schedule_interval="0 0 1 * *",
) as dag:
@task
def empty():
pass
empty()
```
### Operating System
Ubuntu
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with airflow/jobs/scheduler_job_runner.py at the linked lines and review related issue #41014. Reproduce the behavior with the supplied monthly DAG and manual run using the stated logical date. Done means the subsequent scheduled run executes even when a manual run has the same logical date.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100