dotnet / dotnet/macios

[CI] macOS checkout fetch ends with early EOF and loses credentials on retry

Open
#26,385 16 comments 0 reactions 0 assignees View on GitHub
ci-postmortem copilot
Dominant language
C#
Stars
2.9k
Forks
576
Avg merge
2d 13h
Merged PRs (30d)
96

Description

## Summary

Two macOS Sonoma checkout jobs failed before repository build or test tasks with the same Git transfer and retry-authentication sequence.

- Stable signature: `git-fetch-early-eof-credentials-lost`
- Classification: CI infrastructure failure
- Distinct monitored PRs: 2
- Confirmed job occurrences: 2
- First confirmed by the monitor: 2026-07-22T15:59:55Z
- Latest confirmed occurrence: 2026-08-01T03:43:24Z

## Occurrences

- #26235, head `2d6de8eaf12aff5c2f4bff984ce5085b1fece6f7`
- [Build 14731906, attempt 1, X64 Mac Sonoma job](https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=14731906&view=logs&jobId=623bc498-120b-5ff2-984a-5b073efffe1b)
- Checkout failed with an unexpected sideband disconnect, `fatal: early EOF`, and `invalid index-pack output`; task retries then could not read a GitHub username because terminal prompts were disabled.
- Retry outcome: Sonoma attempt 2 succeeded at 2026-07-22T16:29:53Z. The retained timeline also shows a later Sonoma attempt 3 succeeding.
- #26384, head `d45f335fdc821987a0d0086ebd976c6efb545806`
- [Build 14840799, attempt 1, X64 Mac Sonoma job](https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=14840799&view=logs&jobId=623bc498-120b-5ff2-984a-5b073efffe1b)
- Checkout fetched for about 62 minutes, then logged `unexpected disconnect while reading sideband packet`, `fatal: early EOF`, and `fatal: fetch-pack: invalid index-pack output` in task log 2883.
- All task-level retries then failed with `could not read Username for https://github.com: terminal prompts disabled`.
- Build 14840799 completed failed at 2026-08-01T03:47:30Z.

## Reproduction pattern and impact

The macOS checkout performs a large full-ref fetch. When that transfer disconnects after a long download, the retry path reinitializes the repository but no longer has usable GitHub credentials. The job never reaches repository build or test tasks.

## Investigation performed

The Azure build metadata, timelines, checkout task logs, and retry outcomes were inspected. Separate all-state issue searches used the exact sideband-disconnect, early-EOF, invalid-index-pack, credential, and checkout component strings. The only candidate, #10251, concerns product build performance and is unrelated.

No TestSummary or HtmlReport diagnostic is applicable because checkout failed before tests started.

## Retry status

- #26235 recovered on a later Sonoma retry.
- #26384 was not retried. Its completed run also contains deterministic expired-certificate failures, including Windows Integration, and replacement signing credentials are not validated. Retrying the failed set would repeat those failures, and Windows Integration requires an entire-stage retry.

## Current diagnosis and workaround

This is a CI checkout/network failure followed by a broken retry-authentication path. Investigate Git transfer reliability for the macOS agent and ensure checkout retries preserve or reacquire the GitHub authentication header. A fresh checkout attempt can recover, but #26384 must not be retried until the separate signing credentials are validated.

Azure logs and artifacts are retention-limited.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Azure build metadata and checkout task logs for builds 14731906 and 14840799, focusing on the macOS full-ref fetch and its retry path. Trace how the retry handles the GitHub authentication header after early EOF and invalid index-pack errors. Done means the checkout retry preserves or reacquires authentication and the failure pattern is addressed without retrying the unrelated signing-credential failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, git, github, macos
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.