bazelbuild / bazelbuild/bazel

`/dev/shm` exhaustion with async sandbox cleanup

Open
#25,829 5 comments 0 reactions 1 assignee Claimed by @oquenchil View on GitHub
P3 stale team-Local-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

After upgrading to Bazel 8.1.1, I tried removing `--experimental_sandbox_async_tree_delete_idle_threads=0` (which I had been using due to #20965). This resulted in builds filling up `/dev/shm` and then failing (along with other applications on my computer which rely on `/dev/shm` crashing, ie Chrome).

I see the memory usage climb rapidly during a build, and then after stopping it takes around 10 seconds to free the 32G it allocated (or proportionately less if I stop the build before it fills up).

Some `.bazelrc` settings which may be relevant:
```
common --noincompatible_sandbox_hermetic_tmp
common --sandbox_default_allow_network=false
common --incompatible_strict_action_env
common --heap_dump_on_oom
common --sandbox_base=/dev/shm/ --reuse_sandbox_directories
common --watchfs
common --sandbox_fake_hostname=true
common --sandbox_fake_username=true
common --spawn_strategy=linux-sandbox
common --strategy=CopyFile=standalone
common --strategy=CopyDirectory=standalone
common --strategy=TestRunnerSbom=local
```

### Which category does this issue belong to?

Local Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

_No response_

### Which operating system are you running Bazel on?

Debian 12 bookworm

### What is the output of `bazel info release`?

release 8.1.1

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text
Proprietary codebase
```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.