bazelbuild / bazelbuild/bazel

Run docker sandbox without copying inputs

Open
#23,294 5 comments 0 reactions 1 assignee Claimed by @oquenchil View on GitHub
P2 team-Performance type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the feature request:

When some heavy toolchains (eg. toolchains_llvm) are involved, using docker sandbox will experience significant performance degradation due to [copying too many files](https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/sandbox/DockerSandboxedSpawnRunner.java;l=281;bpv=1;bpt=1) around. For example, running a cc_library compilation action will copy 5GB of llvm toolchain into the sandbox exec root.

It should be possible to perform something like [SymlinkedSandboxedSpawn](https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/sandbox/SymlinkedSandboxedSpawn.java;drc=f34d8c925472b481e78d0a4d1dd9447093fbcc0b;bpv=1;bpt=1;l=48), to mount inputs as readonly to the right location, and let docker to manage the rest.

### Which category does this issue belong to?

_No response_

### What underlying problem are you trying to solve with this feature?

Improve the performance of docker sandbox.

### Which operating system are you running Bazel on?

macOS

### What is the output of `bazel info release`?

release 7.2.1

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

Attaching profiles for building openssl with rules_foreign_cc:

- [profile.darwin-sandbox.gz](https://github.com/user-attachments/files/16606671/profile.darwin-sandbox.gz) - running bazel build on darwin with darwin LLVM building for darwin
- [profile.darwin-docker-linux-sandbox.gz](https://github.com/user-attachments/files/16606670/profile.darwin-docker-linux-sandbox.gz) - running bazel build on darwin using linux docker with linux LLVM building for linux (mounted exec root)
- [profile.linux-sandbox.gz](https://github.com/user-attachments/files/16606673/profile.linux.gz) - running bazel build in a linux docker with linux LLVM building for linux (mounted workspace dir, and native exec root)

The overhead for setting up a sandbox for docker is 7~10s for each action.

Also, when running bazel in docker directly the subprocess run time is 100% faster than using docker sandbox (60,234.442 ms vs 131,298.141 ms). I think this is because docker sandbox mounts execroot from host to container instead of letting it uses the native fs. But I didn't push that far to profile the actual run time.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.