bazelbuild / bazelbuild/bazel

bazel should fail if cgroups cannot be created and experimental_sandbox_limits or experimental_sandbox_memory_limit_mb is set

Open
#26,062 14 comments 0 reactions 0 assignees View on GitHub
stale team-Local-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 16h
Merged PRs (30d)
72

Description

### Description of the bug:

I am failing to get `experimental_sandbox_limits` to work to enforce `memory` limits

### Which category does this issue belong to?

Sandboxing?

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

**Setup a cgroup**
```
sudo mkdir /sys/fs/cgroup/example

sudo chown $USER -R /sys/fs/cgroup/example

echo "+memory" | sudo tee /sys/fs/cgroup/example/cgroup.subtree_control
+memory

sudo mkdir /sys/fs/cgroup/example/child

sudo chown $USER -R /sys/fs/cgroup/example/child

echo $$ | sudo tee /sys/fs/cgroup/example/child/cgroup.procs
3477

ps -o cgroup $$
CGROUP
0::/example/child
```

I have a sample program that incrementally allocates memory `//:eat_memory` and a `sh_test` that invokes it.

Here we can see that limiting it to 20 via `experimental_sandbox_limits` does not work ❌
```bash
bazel --experimental_cgroup_parent=/example/child test --spawn_strategy=linux-sandbox --experimental_sandbox_limits=memory=20 //:eat_memory_test --test_output=streamed
WARNING: Option 'local_ram_resources' is deprecated: --local_ram_resources is deprecated, please use --local_resources=memory= instead.
INFO: Invocation ID: 659f47f6-8a45-4501-bf12-e2a1763fe373
WARNING: Option 'local_ram_resources' is deprecated: --local_ram_resources is deprecated, please use --local_resources=memory= instead.
WARNING: Streamed test output requested. All tests will be run locally, without sharding, one at a time
INFO: Analyzed target //:eat_memory_test (0 packages loaded, 0 targets configured).
Running eat_memory test...
Attempting to allocate 2000 MB of memory gradually.
Allocated: 1 MB / 2000 MB
Allocated: 2 MB / 2000 MB
Allocated: 3 MB / 2000 MB
Allocated: 4 MB / 2000 MB
Allocated: 5 MB / 2000 MB
Allocated: 6 MB / 2000 MB
Allocated: 7 MB / 2000 MB
Allocated: 8 MB / 2000 MB
Allocated: 9 MB / 2000 MB
Allocated: 10 MB / 2000 MB
Allocated: 11 MB / 2000 MB
Allocated: 12 MB / 2000 MB
Allocated: 13 MB / 2000 MB
Allocated: 14 MB / 2000 MB
Allocated: 15 MB / 2000 MB
Allocated: 16 MB / 2000 MB
Allocated: 17 MB / 2000 MB
Allocated: 18 MB / 2000 MB
Allocated: 19 MB / 2000 MB
Allocated: 20 MB / 2000 MB
Allocated: 21 MB / 2000 MB
Allocated: 22 MB / 2000 MB
Allocated: 23 MB / 2000 MB
Allocated: 24 MB / 2000 MB
Allocated: 25 MB / 2000 MB
Allocated: 26 MB / 2000 MB
Allocated: 27 MB / 2000 MB
[1 / 2] Testing //:eat_memory_test; 2s linux-sandbox
^C
Bazel caught interrupt signal; cancelling pending invocation.
Allocated: 28 MB / 2000 MB
-- Test timed out at 2025-05-13 18:50:42 UTC --
Target //:eat_memory_test up-to-date:
bazel-bin/eat_memory_test
ERROR: build interrupted
INFO: Elapsed time: 3.136s, Critical Path: 3.01s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully
//:eat_memory_test
```

Okay -- Having looked at this a lot more, it would be great to have diagnostics or log messages if the cgroup fails to get created.

I used https://gist.github.com/fzakaria/6361efe206bcf525e3e6a860c8996049 to check the contents of the cgroup inside a genrule and noticed that even with **linux-sandbox** if the cgroup permissions were not set correctly, the `blaze.slice` would not get created causing the limit toggles to fail.

### Suggestion

If any memory or CPU limit is set and cgroups are failed to be created, Bazel **should error** instead of silently continuing.

### Which operating system are you running Bazel on?
Linux

### What is the output of `bazel info release`?
release 8.2.0

Contributor guide

Open the contributing guide

Research direction

Reproduce the command using --experimental_cgroup_parent, --spawn_strategy=linux-sandbox, and an experimental memory limit, then inspect the linux-sandbox cgroup-creation path and the provided cgroup-checking gist. Done means Bazel reports an error with useful diagnostics when a requested CPU or memory limit cannot create its cgroup, rather than continuing silently.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
build-system, operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.