apache / apache/airflow

Scheduler exception handling in dagrun create loop doesn't work

Open
#59,120 3 comments 0 reactions 0 assignees View on GitHub
area:core area:scheduler kind:bug kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Body

In SchedulerJobRunner, in _create_dag_runs, we do `except Exception: ... continue` but this does not work when the exception is the result of a db error; in that case you need to roll the transaction back.

to fix this we need either smaller transactions (e.g. one per dag run create attempt) or savepoints. either way it's a bit of a refactor.

### Committer

- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start in SchedulerJobRunner._create_dag_runs and inspect how the database transaction spans each DAG run creation attempt. Compare using smaller transactions or savepoints, then verify that a database exception rolls back its transaction and the loop can continue with later DAG runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.