Run docker sandbox without copying inputs
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the feature request:
When some heavy toolchains (eg. toolchains_llvm) are involved, using docker sandbox will experience significant performance degradation due to [copying too many files](https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/sandbox/DockerSandboxedSpawnRunner.java;l=281;bpv=1;bpt=1) around. For example, running a cc_library compilation action will copy 5GB of llvm toolchain into the sandbox exec root.
It should be possible to perform something like [SymlinkedSandboxedSpawn](https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/sandbox/SymlinkedSandboxedSpawn.java;drc=f34d8c925472b481e78d0a4d1dd9447093fbcc0b;bpv=1;bpt=1;l=48), to mount inputs as readonly to the right location, and let docker to manage the rest.
### Which category does this issue belong to?
_No response_
### What underlying problem are you trying to solve with this feature?
Improve the performance of docker sandbox.
### Which operating system are you running Bazel on?
macOS
### What is the output of `bazel info release`?
release 7.2.1
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
_No response_
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
Attaching profiles for building openssl with rules_foreign_cc:
- [profile.darwin-sandbox.gz](https://github.com/user-attachments/files/16606671/profile.darwin-sandbox.gz) - running bazel build on darwin with darwin LLVM building for darwin
- [profile.darwin-docker-linux-sandbox.gz](https://github.com/user-attachments/files/16606670/profile.darwin-docker-linux-sandbox.gz) - running bazel build on darwin using linux docker with linux LLVM building for linux (mounted exec root)
- [profile.linux-sandbox.gz](https://github.com/user-attachments/files/16606673/profile.linux.gz) - running bazel build in a linux docker with linux LLVM building for linux (mounted workspace dir, and native exec root)
The overhead for setting up a sandbox for docker is 7~10s for each action.
Also, when running bazel in docker directly the subprocess run time is 100% faster than using docker sandbox (60,234.442 ms vs 131,298.141 ms). I think this is because docker sandbox mounts execroot from host to container instead of letting it uses the native fs. But I didn't push that far to profile the actual run time.
Contributor guide
Assessment
This issue has not been assessed yet.