GitDagBundle: an interrupted clone can leave untracked working-tree files that block every later checkout
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Apache Airflow Provider(s)
git
### Versions of Apache Airflow Providers
apache-airflow-providers-git 0.4.x (code path unchanged on main)
### Apache Airflow version
3.3.0
### Operating System
Linux (containerized)
### Deployment
Other 3rd-party Helm chart
### Deployment details
Long-lived worker pods (a worker Deployment rather than one pod per task), with the bundle storage path left at its default under `/tmp`, so it is pod-local and lives as long as the pod.
### What happened
A task was marked failed from the UI while its worker was cloning the DAG bundle's tracking repository. The clone started at 06:04:11.472800 and the supervisor sent SIGTERM at 06:04:11.787280, about 315 ms in:
```
{"timestamp":"2026-09-01T06:04:11.775140Z","level":"error","event":"Server indicated the task shouldn't be running anymore","detail":{"reason":"not_running","message":"TI is no longer in the running state and task should terminate","current_state":"failed"},"status_code":409,"logger":"supervisor"}
{"timestamp":"2026-09-01T06:04:11.787280Z","level":"info","event":"Process exited","pid":16,"exit_code":-15,"signal_sent":"SIGTERM","logger":"supervisor"}
```
From then on every task that landed on that worker failed at parse time:
```
Top level error source=task loc=task_runner.py:2347 GitCommandError: Cmd('git') failed due to: exit code(1)
cmdline: git checkout master
stderr: 'error: The following untracked working tree files would be overwritten by checkout:
.agents/instructions/airflow-patterns.md
.agents/instructions/column_naming_conventions.md
.agents/instructions/data-modeling-conventions.md
<...>
Aborting'
File .../airflow/sdk/execution_time/task_runner.py, line 1005 in parse
File .../airflow/providers/git/bundles/git.py, line 244 in initialize
File .../airflow/providers/git/bundles/git.py, line 215 in _initialize
```
Thirty task instances ran on that pod over the following 23 minutes and all thirty failed. Sibling pods in the same ReplicaSet were unaffected, one of them completed 1,180 tasks the same day, and it was the only worker in three weeks with a 100% failure rate. The killed task was the first task that pod ever ran, so it was poisoned on its first bundle initialization and never served a successful task. Recovery only came when the workers rolled to a new ReplicaSet 20 minutes later.
The leftover directory here is different from the one in #72759. There the working tree is empty and the refs are missing, so the checkout fails with `pathspec ... did not match any file(s) known to git`. Here `git clone` had got far enough to write files into the working tree but not far enough to record them in the index, so those files are untracked relative to the incomplete index and the checkout refuses to overwrite them.
Why it never self-heals is the same as in #72759:
1. `_is_pruned_worktree()` treats "no `.git`" as a completed version, and a `.git` is present, so it returns `False`.
2. `_local_repo_has_version()` returns `False` for the truncated repo.
3. `_clone_repo_if_required()` only clones when `repo_path` does not exist. It does exist, so the clone is skipped and the truncated repository is reopened.
4. `self.repo.git.checkout(self.tracking_ref)` then fails. It sits outside both the `try/except` and the `@retry` / `shutil.rmtree` block inside `_clone_repo_if_required()`, so nothing cleans the directory up and nothing retries.
Two things make this worse than a one-off task failure. The failure is raised in `task_runner.parse()`, before the DAG is parsed, so task-level `on_failure_callback` and `on_retry_callback` never run and the failures are invisible to DAG-level alerting. And on a long-lived worker the pod-local bundle directory survives for the pod's lifetime, so a transient fault becomes permanent. On KubernetesExecutor, where every task gets a fresh pod, the same bug is invisible.
### Which of the open fixes cover this
#72787 does. Moving the checkout inside `_clone_repo_if_required()`'s `try` means the `GitCommandError` hits the existing `except (InvalidGitRepositoryError, GitCommandError)` handler, `shutil.rmtree(repo_path)` runs, and the single retry produces a clean clone.
#71535 does not. Fetching the missing tracking refs before the checkout does nothing about untracked files in the working tree, so `git checkout master` still aborts with the error above. So this state is a concrete reason to prefer the broader recovery in #72787 over the narrow fetch, rather than a matter of taste.
### What is still uncovered by either
Both fixes recover from the broken directory after the fact. Neither stops it being created. `Repo.clone_from` still writes straight into `repo_path`, so any interruption can still leave a directory there that the next initialization has to detect and discard. #72759 already noted one residual case this leaves open: a process killed in the narrow window before `git clone` writes the remote configuration leaves a repository with no `origin`, where `self.repo.remotes.origin` raises `AttributeError` rather than anything the recovery path catches.
Cloning into a staging directory next to the final path and renaming it into place once the clone is complete removes the whole class. `rename` is atomic, so `repo_path` either does not exist or holds a finished clone, and an interrupted clone leaves only an orphaned staging directory that the next attempt discards before it starts.
### What you think should happen instead
An interrupted clone should not be able to poison a bundle directory at all, and if a poisoned directory does exist (from an earlier provider version, or from anything else that damaged it), initialization should discard and re-clone it rather than failing forever.
### How to reproduce
The end state reproduces in seconds without racing a real signal:
```bash
git clone --no-checkout tracking_repo # valid .git, empty index and working tree
# copy the repository's tracked files into tracking_repo/ so they are untracked
git -C tracking_repo checkout master
# error: The following untracked working tree files would be overwritten by checkout: ...
# Aborting
```
Point a `GitDagBundle` at that directory and every `initialize()` fails with the traceback above.
For the crash-consistency half, patch `Repo.clone_from` to write a `.git` skeleton into its target and then raise, and check whether `repo_path` exists afterwards. On main it does.
### Anything else
Happy to take the crash-consistency change; I have it working with tests. The recovery half is already covered by #72787 and I would rather that landed than have a competing PR for it.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Research direction
Start in airflow/providers/git/bundles/git.py, especially GitDagBundle.initialize(), _initialize(), and _clone_repo_if_required(), then reproduce the incomplete repository state with the provided git commands. Verify the existing recovery behavior and add tests for an interrupted clone and a damaged directory; done means the final bundle path is never left as a partial clone and initialization can recover from one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100