Difficulties in using Linux-sandbox on Ubuntu 24
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 75
Description
### Description of the bug:
There seems to be a set of issues with "linux-sandbox" strategy when running on Ubuntu 24.10. There were reports on Slack that this was also seen on 24.04 so I am not sure when was this the problem exactly.
## The sandbox is blocked by AppArmor
It seems like the latest AppArmor profile for `unprivileged_userns` will block the `mount` operations by default.
```
sudo dmesg | grep -m1 'DENIED.*linux-sandbox'
[ 91.164199] audit: type=1400 audit(1729776490.609:257): apparmor="DENIED" operation="mount" class="mount" info="failed mntpnt match" error=-13 profile="unprivileged_userns" name="/" pid=4261 comm="linux-sandbox" flags="rw, rprivate"
```
The workaround is to manually create a profile that matches the linux-sandbox binary and allow everything.
```
> cat /etc/apparmor.d/linux-sandbox
# This profile allows everything and only exists to give the
# application a name instead of having the label "unconfined"
abi ,
include
profile linux-sandbox /home//.cache/bazel/_bazel_/install/*/linux-sandbox xattrs=(security.apparmor="trusted") flags=(unconfined) {
userns,
# Site-specific additions and overrides. See local/README for details.
include if exists
}
```
after that, set the xattr with `sudo setfattr -n 'security.apparmor' -v 'trusted' ` for the rule to match the binary.
Finally, reload the AppArmor with `sudo apparmor_parser -r /etc/apparmor.d/linux-sandbox`.
Note that if the path in profile name is not strict enough (i.e. too many glob, not using xattr, etc... ) then it's possible for AppArmor to fallback to `unprivileged_userns` profile instead. To check run `.../linux-sandbox '/bin/true' && sudo dmesg | tail -20`.
I think this is an insane level of gymnastic to set up the host for linux-sandbox to work. Especially when the binary came from Bazel releases, which is managed through Bazelisk. I wonder if we should add something to Bazelisk that would install the AppArmor exclusion automatically?
## `--experimental_use_hermetic_linux_sandbox` does not work.
When you try to run bazel with this flag turn on, you will quickly run into this error message
```
src/main/tools/linux-sandbox-pid1.cc:548: "execvp(/bin/bash, 0x25f93f90)": No such file or directory
```
I think this is caused by `/bin/bash` on ubuntu being a dynamically linked binary
```
> ldd /bin/bash
linux-vdso.so.1 (0x00007b7dd4c50000)
libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007b7dd4a8f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007b7dd4800000)
/lib64/ld-linux-x86-64.so.2 (0x00007b7dd4c52000)
```
so without `/lib/x86_64-linux-gnu/` being available, basic commands won't work.
We can "ask" the user to add these mounts manually... but I think a sane default should be provided when `linux-sandbox` is used?
## `--sandbox_debug` does not work
When troubleshooting all these, I was trying to leverage the debug flag to reproduce the issues.
However, I quickly ran into these errors
```
src/main/tools/linux-sandbox-pid1.cc:98: "open": File exists
...
src/main/tools/linux-sandbox-pid1.cc:110: "link dev/null": File exists
```
This is because the sandbox does not cleanup paths under tmp and dev after each debug runs. So subsequent debug command (copy pasted) would fail to recrate these paths. The workaround here is to clean these paths up manually before each debug copy-paste runs.
```
rm -f //e213eb56b5ca50a8b9a30b5b16a4e504/sandbox/linux-sandbox/11/tmp/empty_file
rm -f //e213eb56b5ca50a8b9a30b5b16a4e504/sandbox/linux-sandbox/11/dev/*
```
I think the linux-sandbox code should tolerate these special files from pre-existing to enable smoother debugging use cases.
### Which category does this issue belong to?
Local Execution
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
_No response_
### Which operating system are you running Bazel on?
Ubuntu 24.10
### What is the output of `bazel info release`?
release 7.4.0
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
_No response_
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
_No response_
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
I think there should be a saner default provided for `linux-sandbox` option. Or we should mark the option as "warning" or "deprecated" to better signal the support status of the technology.
Contributor guide
Assessment
This issue has not been assessed yet.