Refactor backfill_job_runner
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Body
The backfill_job_runner would benefit a lot from refactoring. The gist of it is an 800 line log method that has deep indentations and internal methods, does not use some of the "modernized" ways of doing things (for example retry_db_transaction decorators).
It should be split into smaller logical steps and better documented what it's doing.
That might also help to investigate and fix some of the deadlock issues it is plagued with that we are currently mostly workarounding rather than preventing.
Ideally this could be done in smaller low-risk steps extracting and documenting code piece-by-piece. Might be also a good exercise for someone to learn how it works and document it for others so that this part is quite a bit less mysterious for current and future generations of committers.
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start by locating the backfill_job_runner and reading its 800-line log method to understand the current steps, deep nesting, internal methods, and deadlock workarounds. Compare nearby uses of retry_db_transaction decorators, then plan small, low-risk extractions with documentation. Done means the work is split into logical steps and its behavior and deadlock-related concerns are clearer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100