apache / apache/airflow

Refactor backfill_job_runner

Open
#33,249 8 comments 0 reactions 0 assignees View on GitHub
area:backfill area:core kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Body

The backfill_job_runner would benefit a lot from refactoring. The gist of it is an 800 line log method that has deep indentations and internal methods, does not use some of the "modernized" ways of doing things (for example retry_db_transaction decorators).

It should be split into smaller logical steps and better documented what it's doing.

That might also help to investigate and fix some of the deadlock issues it is plagued with that we are currently mostly workarounding rather than preventing.

Ideally this could be done in smaller low-risk steps extracting and documenting code piece-by-piece. Might be also a good exercise for someone to learn how it works and document it for others so that this part is quite a bit less mysterious for current and future generations of committers.

### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start by locating the backfill_job_runner and reading its 800-line log method to understand the current steps, deep nesting, internal methods, and deadlock workarounds. Compare nearby uses of retry_db_transaction decorators, then plan small, low-risk extractions with documentation. Done means the work is split into logical steps and its behavior and deadlock-related concerns are clearer.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.