[Bug]: Checkpoint refs are written non-atomically — an unclean restart leaves a 0-byte ref that breaks all git fetch/push in the repo

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

T3 writes checkpoint refs to refs/t3/checkpoints/<base64-thread-id>/turn/<n> in the project repo. These appear to be written in place, non-atomically. If the host dies mid-write, the ref file survives the crash with its size committed but its contents never flushed — i.e. as 0 bytes. Git reads a zero-length loose ref as the all-zeros SHA, and from that moment every git command that enumerates local refs fails, which includes fetch, push, and gc.

What happened here:

1. A thread was running and T3 wrote a checkpoint ref for the turn, plus a loose object.

2. ~2 minutes later the host (a Hetzner box running the T3 server) restarted uncleanly.

3. On remount, both files were 0 bytes — the ext4 delayed-allocation crash signature. Both shared the pre-crash mtime and a size of 0:

   0 bytes  .git/refs/t3/checkpoints/<base64-thread-id>/turn/1
   0 bytes  .git/objects/<xx>/<rest-of-sha>
   

4. Every subsequent worktree creation failed, and the background status poller failed on repeat every ~13s.

To reproduce deliberately, truncate any checkpoint ref in a project repo and then create a worktree:

: > .git/refs/t3/checkpoints/<any-existing-thread>/turn/0
git fetch origin        # fatal, before any network access

Expected behavior

A crash mid-checkpoint should not be able to leave the repository in a state where all git remote operations fail. Checkpoint refs should be written atomically (temp file → fsync → rename), which is what git itself does for exactly this reason. At minimum, a zero-length or non-SHA file under refs/t3/** is never legitimate and could be dropped on startup.

Actual behavior

All fetch/push/gc in the affected repo fail until the file is deleted by hand. Worktree creation surfaces:

GitCommandError: Git command failed in GitVcsDriver.fetchRemote (<repo>): git fetch origin failed

This reads as a credentials or connectivity problem and sends you down the wrong path. It is neither — git fetch dies while building its "have" list during negotiation, before opening a socket. The discriminator is git ls-remote, which never walks local refs:

Check	Result
git ls-remote origin HEAD (same URL, creds, network)	succeeds
git rev-list --all (no network at all)	fails, same fatal
DNS + TCP 443 to the forge	OK
Credential helper (gh auth git-credential)	OK
Server process env (HOME, PATH) vs. login shell	identical
git verify-pack	clean

Diagnosing this took a long detour through git fsck, since #4380 means git's actual stderr (fatal: bad ref …) never reaches the user, the logs, or the trace spans. The spans are worse than silent here: GitVcsDriver.fetchRemoteForStatus records exit: {"_tag": "Success"} in server.trace.ndjson even on the polls that failed, so tracing actively points away from the fault.

Related but distinct: #4750 is the same failure class (non-atomic write + crash → NUL/0-byte file bricks a subsystem) for connection-catalog.json; this one is the git ref store. #961 tracks persisted-state corruption generally, but its list of persisted surfaces does not include the repo's refs/t3/**. #5489 is a different checkpoint failure (index.lock contention, intermittent).

Impact

Blocks work completely

Version or commit

0.0.40

Environment

Linux server (ext4, default rw,relatime / data=ordered), git 2.53.0, desktop client on macOS connecting over Tailscale.

Logs or stack traces

Background poller, repeating every ~13s:

GitCommandError: Git command failed in GitVcsDriver.fetchRemoteForStatus (<repo>):
Background Git fetch exited with a non-zero status.
    at statusDetailsRemote (…/t3/dist/bin.mjs)
    at readRemoteStatus …
    at remoteStatus …
    at VcsStatusBroadcaster.refreshRemoteStatus …
    at VcsStatusBroadcaster.retainRemotePoller …

What git actually says, once you run it by hand:

$ git show-ref
fatal: git show-ref: bad ref refs/t3/checkpoints/<…>/turn/1 (0000000000000000000000000000000000000000)

$ git rev-list --all
fatal: bad object refs/t3/checkpoints/<…>/turn/1

$ git fsck --connectivity-only
error: refs/t3/checkpoints/<…>/turn/1: badRefContent:
error: refs/t3/checkpoints/<…>/turn/1: invalid sha1 pointer 0000000000000000000000000000000000000000
error: object file .git/objects/<xx>/<rest-of-sha> is empty

Workaround

Delete the empty ref and object, then re-fetch. Note git update-ref -d will not remove a zero-SHA ref — the file has to be removed directly:

find .git/refs/t3 -type f -size 0 -delete
find .git/objects -type f -size 0 -delete
git fsck --connectivity-only    # expect clean; dangling objects are fine
git fetch --no-tags origin

Any checkpoint whose ref was zeroed is unrecoverable, but it was never a valid commit, so nothing real is lost — that turn just can't be rewound to.

Suggested fixes

1. Write checkpoint refs atomically (temp → fsync → rename). Same for any loose objects T3 writes directly. This is the actual fix.
2. Validate refs/t3/** on startup and drop zero-length / non-SHA ref files.
3. Back off retainRemotePoller after repeated identical failures instead of retrying every ~13s forever.
4. #4380 (carry git's stderr on GitCommandError) would have turned this into a 30-second diagnosis.

Start in apps/server by locating the checkpoint-ref write path and the GitVcsDriver fetch and status-polling entry points. Reproduce the zero-byte ref case, then verify that checkpoint refs are written atomically and that an unclean write cannot leave refs/t3/** able to break fetch, push, or gc.