gitindex silently indexes a fraction of the repository when part of the object store is unreadable by go-git

gitindex writes a shard holding a fraction of the tree and exits 0, with no error and no warning, when part of the object store below the root tree is unreadable by go-git. The shard opens, the repo metadata is right, and searches just return fewer results forever.

Below: git ls-tree -r HEAD lists 50 files, zoekt-git-index indexes 5. We hit this in production and it took a long investigation, because nothing reports it.

Confirmed on main @ eff4bc10cb945064577a64e0faf7ee338698e923. zoekt line numbers are that commit; go-git line numbers are v5.19.2.

(When the commit or root tree itself is unreachable, the build fails loudly instead. That case is fine.)

Reproduce

git init -q A && cd A
git config user.email t@example.com && git config user.name t
mkdir -p dir0{0..9}
for i in $(seq -w 0 49); do echo "package p" > "dir0$((10#$i % 10))/f$i.go"; done
git add -A && git commit -qm base && git repack -a -d -q
cd ..

# Shared clone + one local commit: HEAD, its root tree and the subtree it
# touched are local; the other nine subtrees live only in A, via alternates.
git clone -q --shared A B && cd B
git config user.email t@example.com && git config user.name t
echo "// touched" >> dir00/f00.go && git add -A && git commit -qm touch
cd ..

git -C B ls-tree -r HEAD --name-only | wc -l    # 50
mkdir -p idx && zoekt-git-index -index ./idx ./B && echo "exit=$?"

attempting to index 5 total files (0 via cat-file, 5 via go-git)
finished shard idx/B_v16.00000.zoekt: 3145 index bytes (overhead 12.4), 5 files processed
exit=0

45 of 50 files missing. git reads every one of them (git ls-tree … | while read f; do git cat-file -e "HEAD:$f" || echo BAD; done → nothing). ZOEKT_DISABLE_CATFILE_BATCH=false still reports 5 files: the loss is in the file-list walk, before any blob is read, so the blob backend is irrelevant. And nothing in the API surfaces it — IndexGitRepo returns (bool, error) where the bool means "index was updated".

Cause

CollectFiles treats io.EOF as normal completion — gitindex/tree.go:116-124:

name, entry, err := tw.Next()
if err == io.EOF {
    break
}

go-git's TreeWalker.Next converts a failed subtree load into io.EOF — plumbing/object/tree.go:543-551:

if entry.Mode == filemode.Dir {
    obj, err = GetTree(w.s, entry.Hash)
}
name = simpleJoin(w.base, entry.Name)
if err != nil {
    err = io.EOF          // unreadable subtree == end of walk
    return
}

"Finished" and "hit an object I could not read" are the same signal, so gitindex reports success over a partial list.

Two ways in

Alternates. go-git does check alternates on a miss (object.go:353-366), but resolves the path against the filesystem it was given unless AlternatesFS is set (dotgit.go:1245-1247, :1268-1276). openOptimizedRepo chroots that filesystem to .git and passes only KeepDescriptors (index.go:832-852), and git clone --shared writes an absolute alternates path, which cannot be reached from inside that chroot. Reading a blob that exists only in the alternate:

zoekt openOptimizedRepo (chroot, no alt)   BlobObject: object not found
same + AlternatesFS: osfs.New("/")         OK, size=10
git.PlainOpen                              BlobObject: object not found

So AlternatesFS is both cause and fix. Note line 3: openCompatibleRepo is affected too.

git maintenance. Its loose-objects task writes objects/pack/loose-<hash>.pack, and go-git skips any pack without the pack- prefix (dotgit.go:49, :300). Repos on the incremental strategy pass through this routinely. Which objects land in that pack decides the symptom: on a small fixture it is the tip commit, so you get the visible getCommit("refs/heads/", "HEAD"): reference not found; on a large repo it is deeper subtrees, and you get the silent short index. Use the alternates repro above for the silent case — it is deterministic.

In both cases git reads the repository perfectly. Neither state indicates corruption.

Detecting it

$ git -C B ls-tree -r HEAD --name-only | wc -l
50
$ zoekt -index_dir ./idx -l 'type:file' | wc -l
5

Suggested fixes

1. Do not let an aborted walk look like a finished one. Either get TreeWalker.Next to return the real error upstream, or verify the walk in CollectFiles — e.g. compare the collected count against git ls-tree -r <ref> and fail the build on a shortfall. Failing loudly beats publishing a short index, which is indistinguishable from a small repository.

2. Set AlternatesFS in openOptimizedRepo and openCompatibleRepo: filesystem.Options{KeepDescriptors: true, AlternatesFS: osfs.New("/")} makes the blob above resolve. go-git's own comment at dotgit.go:1280 recommends exactly this.

3. Use SkipReasonMissing in createDocument when the object is genuinely absent, and log it. Today — index.go:1218-1221 — a not-found object is assumed to have been size-filtered:

   // We filter out large documents when fetching the repo. So if an object is too large, it will not be found.
   if errors.Is(err, plumbing.ErrObjectNotFound) {
       return skippedDoc(key, branches, index.SkipReasonTooLarge), nil
   }
   

   When it is absent for any other reason the shard gets the filename with no content — full file coverage, empty index, silently. The cat-file path already does this right (index.go:763-767).

Happy to send a PR for (2) and (3).

Versions: zoekt main @ eff4bc10, also on v0.0.0-20260826170921-df97bab6f7bb; go-git v5.19.2; go1.27.1; git 2.55.0; macOS.

Start with the alternates reproduction, then read CollectFiles in gitindex/tree.go and openOptimizedRepo, openCompatibleRepo, and createDocument in gitindex/index.go. Trace how unreadable subtrees and missing objects are reported. Done means the reproduced repository does not publish a successful partial index, alternates are handled, and genuinely missing objects receive the appropriate skip reason and logging.