bazelbuild / bazelbuild/bazel

worker sandboxing and sandboxed strategy behave differently honour different CLI arguments and are confusing

Open
#17,433 6 comments 0 reactions 1 assignee Assigned to @oquenchil View on GitHub
P2 team-Local-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

The CLI reference states that `--worker_sandboxing` executes workers in a sandbox. Likewise with `--experimental_worker_multiplex_sandboxing` for multiplex workers.

One can also infer that the `sandboxed` strategy will have a _similar_ effect, however given `worker` is also a valid strategy it it confusing how they differ.

It appears that neither `--worker_sandboxing` nor `--experimental_worker_multiplex_sandboxing` are actually effective with the Javac worker.

Out specific use case is with VanillaJavaBuilder; we have found that the worker fills `/tmp` with directories like `/tmp/_tmp18341972703388791833` for each java_library and java_binary target and does not delete them afterwards, eventually running out of space.

The obvious solution is to use `--sandbox_tmpfs_path=/tmp` to mount an ephemeral `/tmp` in the sandbox. This appears only to apply to sandboxes when using the `sandboxed` strategy and not to sandboxes implied by `--worker_sandboxing` and `--experimental_worker_multiplex_sandboxing`.

`--incompatible_sandbox_hermetic_tmp` also appears to be ineffective with `--worker_sandboxing` and `--experimental_worker_multiplex_sandboxing`.

`--sandbox_debug` does not output anything for the implicit worker sandbox either. This makes it difficult to determine if the implicit sandbox is even effective.

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

https://github.com/nickbreen/bazel-repro-sandboxed-malarkey contains a minimum reproduction: caveat it does assume a JDK11 is installed at `opt/jdk11` and ignores the system's JDK.

### Which operating system are you running Bazel on?

Fedora 36

### What is the output of `bazel info release`?

release 6.0.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

N/A

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

```text
N/A
```

### Have you found anything relevant by searching the web?

https://github.com/bazelbuild/bazel/issues/6111 appears to be a similar issue, but has no recent activity and no resolution.

### Any other information, logs, or outputs that you want to share?

I've compared the outputs from these two executions:

```( rm /tmp/_tmp* -r; bazel clean; bazel build //:foo --strategy=Javac=worker; find /tmp/_tmp* ) >sandboxed.out 2>sandboxed.err```

and

```( rm /tmp/_tmp* -r; bazel clean; bazel build //:foo --strategy=Javac=worker; find /tmp/_tmp* ) > worker.out 2>worker.err```

There are only a couple of notable differences:

Sandboxed:
> [0 / 4] [Prepa] BazelWorkspaceStatusAction stable-status.txt
... [common sandbox debug elided] ...
[6 / 7] [Prepa] Building foo.jar (1 source file)
INFO: From Building foo.jar (1 source file):
... [sandbox debut elided] ...
Target //:foo up-to-date:
bazel-bin/foo.jar
bazel-bin/foo
INFO: Elapsed time: 4.285s, Critical Path: 3.22s
INFO: 7 processes: 4 internal, 3 linux-sandbox.
INFO: Build completed successfully, 7 total actions

No `/tmp/_tmp*` directories found.

Worker:

> [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt
... [common sandbox debug elided] ...
INFO: Created new non-sandboxed Javac multiplex-worker (id 2), logging to /home/breenn/.cache/bazel/_bazel_breenn/296a268ebb3259087fc500c19d743bc7/bazel-workers/multiplex-worker-2-Javac.log
[6 / 7] Building foo.jar (1 source file); 0s multiplex-worker
Target //:foo up-to-date:
bazel-bin/foo.jar
bazel-bin/foo
INFO: Elapsed time: 3.599s, Critical Path: 3.28s
INFO: 7 processes: 4 internal, 2 linux-sandbox, 1 worker.
INFO: Build completed successfully, 7 total actions

Files found in tmpdir; e.g. `/tmp/_tmp12060071677398136868/classes/Foo.class`

This log also indicates (after adding `--worker_verbose`) that the worker is not sandboxed despite `--experimental_worker_multiplex_sandboxing`.

Executing without an explicit `--strategy=Javac=` produces the same output with `non-sandboxed Javac`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.