Scheduler exception handling in dagrun create loop doesn't work
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Body
In SchedulerJobRunner, in _create_dag_runs, we do `except Exception: ... continue` but this does not work when the exception is the result of a db error; in that case you need to roll the transaction back.
to fix this we need either smaller transactions (e.g. one per dag run create attempt) or savepoints. either way it's a bit of a refactor.
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start in SchedulerJobRunner._create_dag_runs and inspect how the database transaction spans each DAG run creation attempt. Compare using smaller transactions or savepoints, then verify that a database exception rolls back its transaction and the loop can continue with later DAG runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100