[7.4.1] Bazel Running out of memory with 9GB of heap memory allocated on a runner with 29 GB of ram
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
We're hitting this on bazel 7.4.1 periodically even though our system has 32 GB of memory, container has a 29 GB memory limit. At the time of this error, we were only consuming ~23GB of memory. We're hitting this when running bazel coverage
```
java.util.logging.ErrorManager: 1
java.io.IOException: Cannot allocate memory
at java.base/java.io.FileOutputStream.writeBytes(Native Method)
at java.base/java.io.FileOutputStream.write(Unknown Source)
at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
at java.base/sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
at java.base/sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
at java.base/sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
at java.base/sun.nio.cs.StreamEncoder.lockedWrite(Unknown Source)
at java.base/sun.nio.cs.StreamEncoder.write(Unknown Source)
at java.base/sun.nio.cs.StreamEncoder.write(Unknown Source)
at java.base/java.io.OutputStreamWriter.write(Unknown Source)
at java.base/java.io.Writer.write(Unknown Source)
at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)
at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)
at java.logging/java.util.logging.Logger.log(Unknown Source)
at com.google.common.flogger.backend.system.AbstractBackend.log(AbstractBackend.java:76)
at com.google.common.flogger.backend.system.SimpleLoggerBackend.log(SimpleLoggerBackend.java:31)
at com.google.common.flogger.AbstractLogger.write(AbstractLogger.java:137)
at com.google.common.flogger.LogContext.logImpl(LogContext.java:566)
at com.google.common.flogger.LogContext.log(LogContext.java:681)
at com.google.devtools.build.lib.server.PidFileWatcher.runPidFileChecks(PidFileWatcher.java:91)
at com.google.devtools.build.lib.server.PidFileWatcher.run(PidFileWatcher.java:68)
Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/home/github_actions/.cache/bazel/_bazel_github_actions/049fd0d9a142b0eee346c643b8cf35e6/server/jvm.out')
```
### Which category does this issue belong to?
_No response_
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
no reliable repro
### Which operating system are you running Bazel on?
_No response_
### What is the output of `bazel info release`?
7.4.1
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
_No response_
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
we recently upgraded from 6.5.0 --> 7.3.2 --> 7.4.1 and noticed this occur very frequently on 7.3.2 and 7.4.1
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
We ran bazel coverage on a local executor with a remote cache
Our bazelrc contains the following
```
test:ci --local_resources=gpu_memory_mb=15360 --local_resources=memory=HOST_RAM*0.6
build --remote_download_outputs=all
common --experimental_remote_cache_eviction_retries=5
common --experimental_remote_cache_lease_extension
common --noexperimental_inmemory_dotd_files
common --experimental_worker_for_repo_fetching=off
common --experimental_sandbox_async_tree_delete_idle_threads=0
```
Contributor guide
Research direction
Start with the bazel coverage invocation and the listed --local_resources and remote-cache settings, then trace the failure through PidFileWatcher.java and SimpleLogHandler.java from the stack trace. Establish a reproducible memory-allocation failure and compare behavior across Bazel 6.5.0, 7.3.2, and 7.4.1; done means the cause is identified and the failure no longer occurs under the reported runner limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100