Rebuilds of substitutable dependencies

Some days ago, the Hydra instance I'm using suddenly started doing mass rebuilds of derivations that should be substitutable, despite use-substitutes being enabled. What's more, these builds are not performed as separate build steps. Instead, Hydra only creates steps for a few high-level drvs (which in itself is correct, the triggering commit only affected high-level drvs), and the mass rebuilds are performed by the daemon inside these steps. When I try to build the corresponding drvs using nix-build on the command line, substitution works.

I'm neither exactly sure what happened nor how the build step logic works, but my current best guess is that there was an accidental garbage collection and maybe an invalidation of some of Hydra's GC roots that removed some of the dependencies from the local store. They are however still available in the destination store, a file:// binary cache. Therefore, this check in queue-monitor.c succeeds for almost all drvs, which are therefore not refetched from the binary cache to the local store. I'm only building on localhost, so there is no “send the dependency closure to the build machine” step that might notice this.

I'm probably going to run nix-build for all jobs manually to work around this issue pretty soon – I kind of depend on the builds working – so let me know if there's anything I can do before that to help diagnose the issue.

Start with src/hydra-queue-runner/queue-monitor.cc around line 459 and compare Hydra's localhost build-step dependency handling with the successful nix-build substitution path. Reproduce using the file:// binary cache and inspect when dependencies already in the destination store are fetched. Done means missing substitutable dependencies are restored instead of rebuilt inside high-level steps.