bazelbuild / bazelbuild/bazel

"java.lang.OutOfMemoryError: Java heap space" during repository fetch

Open
#23,029 2 comments 0 reactions 1 assignee Claimed by @Wyverald View on GitHub
P2 team-ExternalDeps type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

We observe the following exception during the repository fetch phase:

```
Fetching ...linux_x86_64__x86_64-unknown-linux-gnu__stable_tools; starting
Fetching ..._tools; Extracting rustc-1.76.0-x86_64-unknown-linux-gnu.tar.x\
z
Fetching repository @@nodejs_linux_amd64; starting
Fetching repository @@npm_typescript; starting
Fetching ...0ae0d/external/npm_typescript; Extracting typescript-5.4.2.tgz
Fetching ...ux_amd64/bin/nodejs; Extracting node-v20.14.0-linux-x64.tar.xz
FATAL: bazel ran out of memory and crashed. Printing stack trace:
net.starlark.java.eval.Starlark$UncheckedEvalError: OutOfMemoryError thrown during Starlark evaluation (LOADING)
at .download_and_extract(:0)
at ._download_node(/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/rules_nodejs/nodejs/repositories.bzl:102)
at ._nodejs_repositories_impl(/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/rules_nodejs/nodejs/repositories.bzl:308)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.tukaani.xz.ArrayCache.getByteArray(Unknown Source)
at org.tukaani.xz.lz.LZDecoder.(Unknown Source)
at org.tukaani.xz.LZMA2InputStream.(Unknown Source)
at org.tukaani.xz.LZMA2Decoder.getInputStream(Unknown Source)
at org.tukaani.xz.BlockInputStream.(Unknown Source)
at org.tukaani.xz.SingleXZInputStream.read(Unknown Source)
at org.tukaani.xz.XZInputStream.read(Unknown Source)
at org.apache.commons.io.IOUtils$$Lambda/0x000000010080a220.apply(Unknown Source)
at org.apache.commons.io.IOUtils.read(IOUtils.java:1958)
at org.apache.commons.io.IOUtils.read(IOUtils.java:1934)
at org.apache.commons.compress.utils.IOUtils.readFully(IOUtils.java:203)
at org.apache.commons.compress.utils.IOUtils.readFully(IOUtils.java:183)
at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.readRecord(TarArchiveInputStream.java:689)
at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getRecord(TarArchiveInputStream.java:497)
at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:415)
at com.google.devtools.build.lib.bazel.repository.CompressedTarFunction.decompress(CompressedTarFunction.java:83)
at com.google.devtools.build.lib.bazel.repository.DecompressorValue.decompress(DecompressorValue.java:131)
at com.google.devtools.build.lib.bazel.repository.starlark.StarlarkBaseExternalContext.downloadAndExtract(StarlarkBaseExternalContext.java:955)
at java.base/java.lang.invoke.LambdaForm$DMH/0x00000001007f3400.invokeVirtual(LambdaForm$DMH)
at java.base/java.lang.invoke.LambdaForm$MH/0x00000001007f5800.invoke(LambdaForm$MH)
at java.base/java.lang.invoke.LambdaForm$MH/0x0000000100530400.invoke(LambdaForm$MH)
at java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Unknown Source)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(Unknown Source)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at net.starlark.java.eval.MethodDescriptor.call(MethodDescriptor.java:178)
at net.starlark.java.eval.BuiltinFunction.fastcall(BuiltinFunction.java:78)
at net.starlark.java.eval.Starlark.fastcall(Starlark.java:806)
at net.starlark.java.eval.Eval.evalCall(Eval.java:682)
at net.starlark.java.eval.Eval.eval(Eval.java:497)
at net.starlark.java.eval.Eval.exec(Eval.java:271)
at net.starlark.java.eval.Eval.execStatements(Eval.java:82)
```

This happens only intermittently. It appears to be a sudden large spike in memory use, because Bazel server memory use up to this point is reasonable and our memory metrics (on 10s collection interval) do not show the process reaching anywhere close to its limit.

Bazel 7.2.0, using `--experimental_worker_for_repo_fetching=off` due to https://github.com/bazelbuild/bazel/issues/22680.

### Which category does this issue belong to?

Core

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Sorry, don't have a minimal example.

### Which operating system are you running Bazel on?

linux

### What is the output of `bazel info release`?

release 7.2.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text
n/a
```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.