Nix fails if my local cache (substituer) is offline. Even when everything is available on the next one: cache.nixos.org

I have many machines on my local network that are using NixOS and they used to be pulling all of their dependencies from the internet (cache.nixos.org).

Since they are all using the almost same configuration, I setup my NAS to act as a local cache:

  nix.settings = {
    substituters = [
      "http://192.168.1.1:5000"
      "https://cache.nixos.org"
    ];
    trusted-public-keys = [
      "192.168.1.1:QwhwNrClkzxCvdA0z3idUyl76Lmho6JTJLWplKtC2ig="
    ];
  };

It works great, saves a lot of time, bandwidth, and resources on cache.nixos.org. I just need to update the NAS first.

My problem is that when the NAS is unavailable, nix stops working on all my machines. Same issue when I use my laptop outside of my local network.

For example:

$ nix shell nixos#konsole 
warning: error: unable to download 'http://192.168.1.1:5000/kbixjq5b2ddnv1vzj01knvrc5j0cbkyv.narinfo': Couldn't connect to server (7); retrying in 307 ms
warning: error: unable to download 'http://192.168.1.1:5000/kbixjq5b2ddnv1vzj01knvrc5j0cbkyv.narinfo': Couldn't connect to server (7); retrying in 520 ms
warning: error: unable to download 'http://192.168.1.1:5000/kbixjq5b2ddnv1vzj01knvrc5j0cbkyv.narinfo': Couldn't connect to server (7); retrying in 1195 ms
warning: error: unable to download 'http://192.168.1.1:5000/kbixjq5b2ddnv1vzj01knvrc5j0cbkyv.narinfo': Couldn't connect to server (7); retrying in 2116 ms
error: unable to download 'http://192.168.1.1:5000/kbixjq5b2ddnv1vzj01knvrc5j0cbkyv.narinfo': Couldn't connect to server (7)

and the command fails without installing konsole.

Then if I make the NAS available again, nix will successfully see that konsole was not present on the NAS and use cache.nixos.org instead.

This means that the logic to "try the next substituer" is already there, but only works when the error on the first one is a 404 but not when it's a failed connection.

I'd be happy to make a PR if someone could give me a pointer or two about where the offending code is.

Start by reproducing the failure with nix shell nixos#konsole while the first substituter is unreachable. Trace the substituter fallback path and verify that a connection failure advances to cache.nixos.org instead of aborting; done means the command succeeds using the next available cache.