Outboard toolchains

The problem

1. Remote builds need hermetic toolchains.
2. Toolchains are much like regular build rules and can be built on-demand.[^1] Projects like rules_nixpkgs exploit this to great effect.
3. Toolchains often need absolute paths. Nix especially so.[^2]
4. However, fixed, absolute paths are completely incompatible with RBE caching. If your toolchain is built by bazel/buck, but must be placed in /nix/store to execute properly, then it can't be tracked / uploaded / downloaded by an executor. So Nix toolchains only work on local machines. So much for hermeticity.

(4) makes things very impractical for Nix-based toolchains, and the only viable solution right now is not to build toolchains on-demand at all. Instead, you must describe every nix dependency you need in advance, package it all up into a Docker image (example), and re-upload the image every time you want to add a tool. When you have a few GB of dependencies, my experience with a similar pattern is that while you can do it, this slows down delivering tooling improvements to a snail's pace. (It is always feasible if you have a "developer platforms team" or similar at a big company. But the Nix project is your "developer platforms team" at a small company.)

Some links:

• My comment on https://github.com/facebook/buck2/pull/466#issuecomment-1809544301 summarising some approaches
• A rundown of the Docker images approach taken by rules_ll, in a long, long thread of other people with this exact problem https://github.com/tweag/rules_nixpkgs/issues/180#issuecomment-1526940723

Why should Native Link help?

I believe this is a pretty signifcant barrier to adopting remote builds. Anyone can use bazel and buck without RBE, it's child's play. But getting a remote-compatible toolchain together is pretty hard, and takes a lot of effort. At my employer, this has almost singlehandedly blocked the adoption of Buck -- the benefits of RBE would take too much time investment to obtain, so the whole thing is not worth it. Anything you can offer to make this work would be a big differentiator.

How can Native Link help?

You're enterprising folks, and you seem to be using Nix yourselves. Do you have any ideas to make on-demand Nix toolchains work? I have two:

1. Absolute output_paths

Is there perhaps a way to bring /nix/store back into the cache? I can imagine a possible extension to the RBE protocol, to support absolute paths in action cache output paths, moderated by a whitelist of /nix/store for example. Tricky because all Nix actions would write into the same directory. If every Nix "store path" were a separate action in the bazel/buck build graph, and you had to provide the actual path in e.g. nix_package(name = ..., expr = "pkgs.coreutils", path = "/nix/store/hcil3fgcjav0y458ff4m98zgcqky00gk-coreutils-9.3"), then this would be doable, especially if you could autogenerate rules like this.

2. Resolver for symlinks to things in /nix/store

Alternatively, there could be special treatment of symlinks to things in /nix/store or any other configurable whitelisted path. That would not require modifying RBE clients for support, because they can all send symlinks already. It wouldn't even require changing rules_nixpkgs, because it already uses symlinks to /nix/store, placed in the bazel output. (I have still not released my version of rules_nixpkgs for buck, but it too works this way.)

The idea involves a symlink resolver, which would attempt to pre-resolve symlinks of some given pattern by just hitting a Nix remote cache. The resolver would run before actions that depend on that symlink execute. This could be configured like so:

• a whitelist of paths like /nix/store to have a resolver run for, like { "/nix/store": ["./nix_resolver.sh"] }
• a resolver, i.e. a command line program that is run with any paths to fetch as input. Nix users will want this to run e.g. nix-store --realize /nix/store/hcil3fgcjav0y458ff4m98zgcqky00gk-coreutils-9.3 for each command line argument. Obviously you can configure your own nix remote cache to hit, etc, in the shell script. The resolver can output a list of paths to be hardlinked alongside the rest of the action inputs. (Hardlink? Not sure. Although if Nix is going to be running on the worker nodes in order to resolve these things, might want to let Nix manage the /nix directory itself, copy resolved paths to the NL cache, and then expose those cached paths through some other means, like bind mounts. I'm thinking this would be handled by an environment variable provided to the execution script, and so people can choose to use docker volume mounts to make the paths available.)
• a cache configuration for resolved paths. You probably don't want to hit a remote store every time you run an action with sh in it, when sh could be stored in the regular cache hierarchy and be treated like any other file.

This would play really nicely with local builds, because your local machine's rules_nixpkgs or equivalent will just be realizing nix paths from a nix cache, symlinking those paths into the bazel/buck output directory, and adding those symlinks as Nix GC roots so that nix store gc doesn't delete them. This will end up having the exact same effect when it comes to remote execution, just that the symlinks will be made to work a different way.

[^1]: Say you've got a repo with a frontend and a backend. A backend developer might never trigger the toolchain rules that download Node.js in order to build a JS bundle for the frontend.
[^2]: Installed nix packages live in /nix/store, and contain many absolute path references to other things in /nix/store. This is just a slightly more upfront incarnation of a problem that exists elsewhere; Nix just fails very very fast when it's in the wrong place. All dynamically linked libraries are referenced via absolute paths in ELF files, for example.