worker sandboxing and sandboxed strategy behave differently honour different CLI arguments and are confusing
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
The CLI reference states that `--worker_sandboxing` executes workers in a sandbox. Likewise with `--experimental_worker_multiplex_sandboxing` for multiplex workers.
One can also infer that the `sandboxed` strategy will have a _similar_ effect, however given `worker` is also a valid strategy it it confusing how they differ.
It appears that neither `--worker_sandboxing` nor `--experimental_worker_multiplex_sandboxing` are actually effective with the Javac worker.
Out specific use case is with VanillaJavaBuilder; we have found that the worker fills `/tmp` with directories like `/tmp/_tmp18341972703388791833` for each java_library and java_binary target and does not delete them afterwards, eventually running out of space.
The obvious solution is to use `--sandbox_tmpfs_path=/tmp` to mount an ephemeral `/tmp` in the sandbox. This appears only to apply to sandboxes when using the `sandboxed` strategy and not to sandboxes implied by `--worker_sandboxing` and `--experimental_worker_multiplex_sandboxing`.
`--incompatible_sandbox_hermetic_tmp` also appears to be ineffective with `--worker_sandboxing` and `--experimental_worker_multiplex_sandboxing`.
`--sandbox_debug` does not output anything for the implicit worker sandbox either. This makes it difficult to determine if the implicit sandbox is even effective.
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
https://github.com/nickbreen/bazel-repro-sandboxed-malarkey contains a minimum reproduction: caveat it does assume a JDK11 is installed at `opt/jdk11` and ignores the system's JDK.
### Which operating system are you running Bazel on?
Fedora 36
### What is the output of `bazel info release`?
release 6.0.0
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
N/A
### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?
```text
N/A
```
### Have you found anything relevant by searching the web?
https://github.com/bazelbuild/bazel/issues/6111 appears to be a similar issue, but has no recent activity and no resolution.
### Any other information, logs, or outputs that you want to share?
I've compared the outputs from these two executions:
```( rm /tmp/_tmp* -r; bazel clean; bazel build //:foo --strategy=Javac=worker; find /tmp/_tmp* ) >sandboxed.out 2>sandboxed.err```
and
```( rm /tmp/_tmp* -r; bazel clean; bazel build //:foo --strategy=Javac=worker; find /tmp/_tmp* ) > worker.out 2>worker.err```
There are only a couple of notable differences:
Sandboxed:
> [0 / 4] [Prepa] BazelWorkspaceStatusAction stable-status.txt
... [common sandbox debug elided] ...
[6 / 7] [Prepa] Building foo.jar (1 source file)
INFO: From Building foo.jar (1 source file):
... [sandbox debut elided] ...
Target //:foo up-to-date:
bazel-bin/foo.jar
bazel-bin/foo
INFO: Elapsed time: 4.285s, Critical Path: 3.22s
INFO: 7 processes: 4 internal, 3 linux-sandbox.
INFO: Build completed successfully, 7 total actions
No `/tmp/_tmp*` directories found.
Worker:
> [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt
... [common sandbox debug elided] ...
INFO: Created new non-sandboxed Javac multiplex-worker (id 2), logging to /home/breenn/.cache/bazel/_bazel_breenn/296a268ebb3259087fc500c19d743bc7/bazel-workers/multiplex-worker-2-Javac.log
[6 / 7] Building foo.jar (1 source file); 0s multiplex-worker
Target //:foo up-to-date:
bazel-bin/foo.jar
bazel-bin/foo
INFO: Elapsed time: 3.599s, Critical Path: 3.28s
INFO: 7 processes: 4 internal, 2 linux-sandbox, 1 worker.
INFO: Build completed successfully, 7 total actions
Files found in tmpdir; e.g. `/tmp/_tmp12060071677398136868/classes/Foo.class`
This log also indicates (after adding `--worker_verbose`) that the worker is not sandboxed despite `--experimental_worker_multiplex_sandboxing`.
Executing without an explicit `--strategy=Javac=` produces the same output with `non-sandboxed Javac`.
Contributor guide
Assessment
This issue has not been assessed yet.