buildkite / buildkite/agent

Git mirror branch ref goes permanently stale when a tag shares the branch name

Open
#4,211 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
1.1k
Forks
378
Avg merge
2d 6h
Merged PRs (30d)
74

Description

## Summary

When a tag shares a branch's name, the git mirror's warm-path update fetches the **tag** instead of the branch, and the mirror's `refs/heads/` silently stays stale — on every job, forever, until the tag is deleted or the mirror is recreated.

## Cause

`updateGitMirror` fetches the build branch by bare name:

```
git --git-dir fetch origin
```

Git resolves a bare refspec source against `refs/tags/` before `refs/heads/`. With a tag named the same as the branch, the fetch succeeds (exit 0), reports `* tag -> FETCH_HEAD`, and never updates `refs/heads/` in the mirror. The mirror update relies on the `--mirror` config's `+refs/*:refs/*` refspec to opportunistically update the branch ref from the fetched refs — but the fetched ref is the tag, so the branch ref is untouched.

## Reproduction (plain git, no agent needed)

```bash
export GIT_AUTHOR_NAME=a GIT_AUTHOR_EMAIL=a@b GIT_COMMITTER_NAME=a GIT_COMMITTER_EMAIL=a@b
cd "$(mktemp -d)"
git init -q --bare canon.git
git clone -q canon.git work && cd work
git commit -q --allow-empty -m a; A=$(git rev-parse HEAD)
git commit -q --allow-empty -m b
git branch -m release && git push -q origin release
git checkout -qb tagline "$A" && git commit -q --allow-empty -m t
git tag release && git push -q origin refs/tags/release
cd .. && git clone -q --mirror canon.git mirror.git
cd work && git checkout -q release && git commit -q --allow-empty -m c
C=$(git rev-parse HEAD)
git push -q origin refs/heads/release:refs/heads/release
cd ../mirror.git
git fetch origin release # what the agent's mirror update runs
echo "canonical tip: $C"
echo "mirror branch: $(git rev-parse refs/heads/release)"
```

Output:

```
warning: refname 'release' is ambiguous.
From .../canon
* tag release -> FETCH_HEAD
canonical tip: d983dfb1...
mirror branch: 93ad5a7c... <-- stale, never advances
```

## Impact

- The mirror still ends up containing the needed *objects* in most cases (the checkout's own fetch of the commit heals the job), so builds generally pass — which is why this stays invisible.
- But the mirror's branch ref drifts permanently, so anything that consumes mirror refs (reference clones resolving the branch, `--git-mirrors-skip-update` setups, humans inspecting the mirror) sees an old tip.
- Branch/tag name collisions are common in release workflows (e.g. a `v1` maintenance branch and a `v1` tag, or `release` branch + `release` tag).

A related shape of the same problem: the bare refspec is also passed through `shellwords.Split`, which mangles branch names containing legal-but-unusual characters (quotes are valid in refnames).

## Fix

Fetch the branch by explicit forced refspec as a single argument:

```
git --git-dir fetch -- origin "+refs/heads/:refs/heads/"
```

This is implemented as part of #4210 (where the stale ref would otherwise have become a verification-soundness issue), including a regression test (`TestUpdateGitMirrorBranchTipFreshDespiteSameNamedTag`) that fails against the bare-name fetch. Tag builds — where `BUILDKITE_BRANCH` may legitimately name the tag — keep the bare-name fetch.

Filing this separately so the underlying bug is tracked and searchable independently of that PR; if #4210 lands, this can be closed with it.

Contributor guide

Open the contributing guide

Research direction

Start at updateGitMirror and review the named regression test, TestUpdateGitMirrorBranchTipFreshDespiteSameNamedTag. Run that test against the current behavior and verify that a branch ref advances when a same-named tag exists, while tag builds retain their existing behavior; the issue notes that this is implemented as part of #4210.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, go
Domain
ci-cd, devops
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.