bazelbuild / bazelbuild/bazel

[7.5.0] gRPC RESOURCE_EXHAUSTED when linking large binaries

Open
#27,124 2 comments 0 reactions 0 assignees View on GitHub
awaiting-user-response team-Remote-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

We recently ran into an issue where we hit the gRPC limit for message size when building a binary that has many files. We use remote execution/caching. We fixed this locally on a fork by upping the limit size - https://github.com/mongodb-forks/bazel/commit/06d753863dde251110daef739d2c3e419782b881 - but it seems like from other similar bugs Bazel prefers to fix this by chunking the messages rather than upping the limit.

```
[2025/09/30 15:12:23.898] ERROR: /data/mci/a505593e9fd0b91b5839bcaafff99d8f/src/BUILD.bazel:278:14 Writing: bazel-out/aarch64-opt/bin/dist-test-stripped.zst failed: (Exit 34): RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 4199215
[2025/09/30 15:12:23.898] java.io.IOException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 4199215
[2025/09/30 15:12:23.898] at com.google.devtools.build.lib.remote.GrpcCacheClient.lambda$handleStatus$5(GrpcCacheClient.java:265)
[2025/09/30 15:12:23.898] at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:205)
[2025/09/30 15:12:23.898] at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:192)
[2025/09/30 15:12:23.898] at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:134)
[2025/09/30 15:12:23.898] at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[2025/09/30 15:12:23.898] at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
[2025/09/30 15:12:23.898] at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
[2025/09/30 15:12:23.898] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
[2025/09/30 15:12:23.898] at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:105)
[2025/09/30 15:12:23.899] at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[2025/09/30 15:12:23.899] at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
[2025/09/30 15:12:23.899] at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
[2025/09/30 15:12:23.899] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
[2025/09/30 15:12:23.899] at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:55)
[2025/09/30 15:12:23.899] at com.google.devtools.build.lib.remote.util.RxFutures$2.onError(RxFutures.java:259)
[2025/09/30 15:12:23.899] at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:117)
[2025/09/30 15:12:23.899] at io.reactivex.rxjava3.internal.operators.single.SingleUsing$UsingSingleObserver.onError(SingleUsing.java:180)
[2025/09/30 15:12:23.899] at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.tryOnError(SingleCreate.java:95)
[2025/09/30 15:12:23.899] at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:81)
[2025/09/30 15:12:23.899] at com.google.devtools.build.lib.remote.util.RxFutures$OnceSingleOnSubscribe$1.onFailure(RxFutures.java:172)
[2025/09/30 15:12:23.899] at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1127)
[2025/09/30 15:12:23.899] at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[2025/09/30 15:12:23.899] at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
[2025/09/30 15:12:23.899] at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
[2025/09/30 15:12:23.899] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
[2025/09/30 15:12:23.899] at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:572)
[2025/09/30 15:12:23.899] at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:542)
[2025/09/30 15:12:23.900] at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
[2025/09/30 15:12:23.900] at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
[2025/09/30 15:12:23.900] at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
[2025/09/30 15:12:23.900] at com.google.devtools.build.lib.remote.NetworkTimeInterceptor$NetworkTimeCall$1.onClose(NetworkTimeInterceptor.java:81)
[2025/09/30 15:12:23.900] at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
[2025/09/30 15:12:23.900] at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
[2025/09/30 15:12:23.900] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
[2025/09/30 15:12:23.900] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
[2025/09/30 15:12:23.900] at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[2025/09/30 15:12:23.900] at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
[2025/09/30 15:12:23.900] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[2025/09/30 15:12:23.900] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[2025/09/30 15:12:23.900] at java.base/java.lang.Thread.run(Unknown Source)
[2025/09/30 15:12:23.900] Caused by: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 4199215
```

### Which category does this issue belong to?

Remote Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

In our case cc_binary was linking thousands of very long path c++ files, don't have a minimal repro.

### Which operating system are you running Bazel on?

Linux

### What is the output of `bazel info release`?

7.5.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text

```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

_No response_

### Have you found anything relevant by searching the web?

Found this - https://github.com/bazelbuild/bazel/issues/24705 - i think this is a different area where it hits the limit though

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the remote execution failure path at com.google.devtools.build.lib.remote.GrpcCacheClient.java:265 and investigate how large link results are sent through gRPC. Compare the reported 4 MiB limit with the linked fork change and the related issue 24705. Done means large remote-execution link results no longer fail with RESOURCE_EXHAUSTED; no minimal reproduction or test file is provided.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
build-system, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.