facebook / facebook/buck2

RE: Error, message length too large for `BatchReadBlobs`

Open
#583 1 comment 2 reactions 0 assignees View on GitHub
bug remote execution
Dominant language
Rust
Stars
4.4k
Forks
394
PR merge metrics
No merged PRs in 30d

Description

We are using the bazel-remote-worker locally for isolation.

For a target we see this error reproducibly:
```
[2024-02-13T10:34:08.702+00:00] Action failed: root//frontend:app (prelude//platforms:default#213ed1b7ab869379) (genrule)
[2024-02-13T10:34:08.702+00:00] Internal error (stage: materialize_outputs): action_digest=9f86841c1d695a3dbf441d2d9c984a6c750f2ccbd6b0e59de286f42e5ff060ab:142: Failed to declare in materializer: Failed to make BatchReadBlobs request: status: OutOfRange, message: "Error, message length too large: found 4194593 bytes, the limit is: 4194304 bytes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "grpc-encoding": "identity", "grpc-accept-encoding": "gzip"} }
```
I found this [post](https://cprimozic.net/notes/posts/rust-tonic-request-response-size-limits/), that pin pointed the problem. This error is actually thrown by tonic, [here](https://github.com/hyperium/tonic/blob/1934825ff52bff26bb88b709aee9ac73d3ea51c0/tonic/src/codec/decode.rs#L184).

The bazel-remote-worker reports a max batch size of 4MiB in its capabilities. If a request's batch size is getting close to this limit, the server response (including headers and maybe compression meta data) might exceed the 4MiB default transport message size enforced by tonic and cause this error.

As a workaround we have set the max batch size to 4MB for the bazel-remote-worker.

The max size enforced by tonic is also configurable and probably should be set to a value larger than the max batch size reported by the server.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.