bazelbuild / bazelbuild/bazel

[7.4.1] Bazel Running out of memory with 9GB of heap memory allocated on a runner with 29 GB of ram

Open
#24,527 34 comments 0 reactions 0 assignees View on GitHub
P2 team-Performance type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

We're hitting this on bazel 7.4.1 periodically even though our system has 32 GB of memory, container has a 29 GB memory limit. At the time of this error, we were only consuming ~23GB of memory. We're hitting this when running bazel coverage
```
java.util.logging.ErrorManager: 1
java.io.IOException: Cannot allocate memory
at java.base/java.io.FileOutputStream.writeBytes(Native Method)
at java.base/java.io.FileOutputStream.write(Unknown Source)
at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
at java.base/sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
at java.base/sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
at java.base/sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
at java.base/sun.nio.cs.StreamEncoder.lockedWrite(Unknown Source)
at java.base/sun.nio.cs.StreamEncoder.write(Unknown Source)
at java.base/sun.nio.cs.StreamEncoder.write(Unknown Source)
at java.base/java.io.OutputStreamWriter.write(Unknown Source)
at java.base/java.io.Writer.write(Unknown Source)
at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)
at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)
at java.logging/java.util.logging.Logger.log(Unknown Source)
at com.google.common.flogger.backend.system.AbstractBackend.log(AbstractBackend.java:76)
at com.google.common.flogger.backend.system.SimpleLoggerBackend.log(SimpleLoggerBackend.java:31)
at com.google.common.flogger.AbstractLogger.write(AbstractLogger.java:137)
at com.google.common.flogger.LogContext.logImpl(LogContext.java:566)
at com.google.common.flogger.LogContext.log(LogContext.java:681)
at com.google.devtools.build.lib.server.PidFileWatcher.runPidFileChecks(PidFileWatcher.java:91)
at com.google.devtools.build.lib.server.PidFileWatcher.run(PidFileWatcher.java:68)

Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/home/github_actions/.cache/bazel/_bazel_github_actions/049fd0d9a142b0eee346c643b8cf35e6/server/jvm.out')
```

### Which category does this issue belong to?

_No response_

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

no reliable repro

### Which operating system are you running Bazel on?

_No response_

### What is the output of `bazel info release`?

7.4.1

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

_No response_

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

we recently upgraded from 6.5.0 --> 7.3.2 --> 7.4.1 and noticed this occur very frequently on 7.3.2 and 7.4.1

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

We ran bazel coverage on a local executor with a remote cache

Our bazelrc contains the following

```
test:ci --local_resources=gpu_memory_mb=15360 --local_resources=memory=HOST_RAM*0.6

build --remote_download_outputs=all
common --experimental_remote_cache_eviction_retries=5
common --experimental_remote_cache_lease_extension
common --noexperimental_inmemory_dotd_files
common --experimental_worker_for_repo_fetching=off
common --experimental_sandbox_async_tree_delete_idle_threads=0
```

Contributor guide

Open the contributing guide

Research direction

Start with the bazel coverage invocation and the listed --local_resources and remote-cache settings, then trace the failure through PidFileWatcher.java and SimpleLogHandler.java from the stack trace. Establish a reproducible memory-allocation failure and compare behavior across Bazel 6.5.0, 7.3.2, and 7.4.1; done means the cause is identified and the failure no longer occurs under the reported runner limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.