`UNAVAILABLE` RBE cache/keepalive fail
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 430
Description
Since migrating to Engflow RBE we see an issue with bazel keepalives failing (afaict) trying to upload/download large outputs
Ive had a few different theories about what is causing this but none seem to have proven true so far
One clue is that altho it happens fairly frequently everywhere (including in mobile where its been an issue for a while) it seems to happen most freqently in our arm build - which is different in that it both doesnt use remote execution (ie cache only) and it has a lot more cpus/memory
This is probably our number one cause of CI failure at this point
typical error:
```console
ERROR: /source/distribution/binary/BUILD:37:8: Writing: bazel-out/aarch64-opt/bin/distribution/binary/release.tar.zst failed: Exec failed due to IOException: 3 errors during bulk transfer:
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: Keepalive failed. The connection is likely gone
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: Keepalive failed. The connection is likely gone
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: Keepalive failed. The connection is likely gone
```
i tried adjusting the `grpc_keepalive_time` and `grpc_keepalive_timeout` flags (upwards) but that does not seem to have helped
Contributor guide
Assessment
This issue has not been assessed yet.