envoyproxy / envoyproxy/envoy

`UNAVAILABLE` RBE cache/keepalive fail

Open
#36,924 6 comments 0 reactions 0 assignees View on GitHub
area/ci bug no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

Since migrating to Engflow RBE we see an issue with bazel keepalives failing (afaict) trying to upload/download large outputs

Ive had a few different theories about what is causing this but none seem to have proven true so far

One clue is that altho it happens fairly frequently everywhere (including in mobile where its been an issue for a while) it seems to happen most freqently in our arm build - which is different in that it both doesnt use remote execution (ie cache only) and it has a lot more cpus/memory

This is probably our number one cause of CI failure at this point

typical error:

```console

ERROR: /source/distribution/binary/BUILD:37:8: Writing: bazel-out/aarch64-opt/bin/distribution/binary/release.tar.zst failed: Exec failed due to IOException: 3 errors during bulk transfer:
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: Keepalive failed. The connection is likely gone
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: Keepalive failed. The connection is likely gone
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: Keepalive failed. The connection is likely gone
```

i tried adjusting the `grpc_keepalive_time` and `grpc_keepalive_timeout` flags (upwards) but that does not seem to have helped

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.