Known issues with git mirrors

This issue exists to billboard known problems with git mirrors across different agent versions.

## Checkouts sometimes fail with 'bad object', 'did not send all necessary objects' errors

**Issue:** Checkouts sometimes fail because the mirror appears to contain the commit needed for a job, but the checkout phase fails with a 'bad object' or '{repo} did not send all necessary objects' error. As a recovery step the agent may try to fetch _all_ refs, but still fails again. This could be due to corruption in the mirror repo, or due to objects removed normally from the mirror but still referenced by the checkout directory.

**Affected versions:** Unknown, possibly since v3.23.0.

**Status:** A workaround (#2286) is in v3.52.0.

**Workarounds/debugging:**

- Update to the latest agent release.
- Use the `--clean-checkout` flag or set the environment variable `BUILDKITE_CLEAN_CHECKOUT=true`.
- Delete the affected mirror directory, allowing the agent to make a fresh clone.
- Manually run `git fsck --full` in the checkout directory and manually remove any bad objects.
- In the mirror, manually run `git fsck --full`, remove bad objects, and then run `git remote update` to ensure the mirror is not corrupt.
- Disable git mirrors.

## Mirrors of submodules not fetched

**Issue:** When the agent runs jobs that check out a repository that uses git submodules, the submodules should also be mirrored. The main repository mirror is updated (fetched) to ensure the commit or branch being built was present, but a bug meant that submodule mirrors was not fetched. As a result, submodule mirrors might become out -of-date, missing commits or objects needed by the main repo.

**Affected versions:** Between v3.44.0 and v3.49.0, inclusive.

**Status:** Fixed with #2203, which was released as part of v3.50.

**Workarounds/debugging:**

- Update to the latest agent release.
- If each host only runs one agent, include a manual `git fetch origin` for all submodule mirror directories in your pipeline.
- Use a custom checkout hook.
- Disable git mirrors.

## Mirrors of submodules pull from wrong remote URL

**Issue:** When the agent runs jobs that check out a repository that uses git submodules, the submodules should also be mirrored. A bug caused the remote URL in the mirror directory for each submodule to be the wrong URL.

**Affected versions:** Between v3.44.0 and v3.48.0, inclusive.

**Status:** Fixed with #2144, but because the submodule mirror might not be re-fetched, it can still contain the wrong git history and objects. The missing fetch is fixed by #2203, which was released as part of v3.50.

**Workarounds/debugging:**

- Update to the latest agent release.
- If each host only runs one agent, update to at least agent v3.49 and include a manual `git fetch origin` for all submodule mirror directories in your pipeline.
- Use a custom checkout hook.
- Disable git mirrors.

No files or tests are named; start by reviewing the issue's documented mirror failures and the referenced fixes #2286, #2203, and #2144. The issue contains workarounds and version statuses but does not define a remaining documentation change, so completion criteria would need to be established first.

Written by the indexing model from the issue text.