bazelbuild / bazelbuild/bazel

Remote cache server failure causes build to fail with internal error

Open
#19,184 2 comments 1 reaction 1 assignee Claimed by @coeuvre View on GitHub
P2 team-Remote-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

A build was recently failed with an internal error (exit code 37). This error was caused by a problem with the remote cache server. The expected behavior is for the build to fail with an exit code that indicates the cache problem.

### Which category does this issue belong to?

Remote Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Provided stack trace can provide information about the steps that led to a bug, which can be used to reproduce the bug.

### Which operating system are you running Bazel on?

Windows 10

### What is the output of `bazel info release`?

release 6.1.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

_No response_

### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

From Bazel server logs:
```
Error evaluating artifact nested set. First exception: com.google.devtools.build.lib.actions.AlreadyReportedActionExecutionException: Exec failed due to IOException: 426 errors during bulk transfer:
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: ***
...
230804 02:01:04.404:XT 34 [com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively] Shutting down due to exception
java.lang.IllegalStateException: Null value for mandatory [[]bazel-out/***.obj (ActionLookupData{actionLookupKey=ConfiguredTargetKeyjava.lang.IllegalStateException: Null value for mandatory [[]bazel-out/***.obj (ActionLookupData{actionLookupKey=ConfiguredTargetKey{label=//***:***, config=BuildConfigurationKey[3ecf50d75eaa6ba86d1580b810ca96c70236542c03cca111901344a1c0b72cd0]}, actionIndex=2}) with no errors or values missing: ActionLookupData{actionLookupKey=ConfiguredTargetKey{label=//***:***, config=BuildConfigurationKey[3ecf50d75eaa6ba86d1580b810ca96c70236542c03cca111901344a1c0b72cd0]}, actionIndex=15} action 'Linking ***.lib'
Action does not discover inputs
Input is an input of action, bottom-up path:
11, [File:[[]bazel-out/***.obj, File:[[]bazel-out/***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj]
49, [File:[[]bazel-out/***.obj, File:[[]bazel-out/***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj, File:[[]bazel-out/x64_windows-opt-***/bin]***.obj]

at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.accumulateInputs(ActionExecutionFunction.java:1226)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkInputs(ActionExecutionFunction.java:1060)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:272)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:161)
at com.google.devtools.build.skyframe.ParallelEvaluator.bubbleErrorUp(ParallelEvaluator.java:427)
at com.google.devtools.build.skyframe.ParallelEvaluator.waitForCompletionAndConstructResult(ParallelEvaluator.java:216)
at com.google.devtools.build.skyframe.ParallelEvaluator.doMutatingEvaluation(ParallelEvaluator.java:182)
at com.google.devtools.build.skyframe.ParallelEvaluator.eval(ParallelEvaluator.java:677)
at com.google.devtools.build.skyframe.InMemoryMemoizingEvaluator.evaluate(InMemoryMemoizingEvaluator.java:203)
at com.google.devtools.build.lib.skyframe.SkyframeExecutor.buildArtifacts(SkyframeExecutor.java:1514)
at com.google.devtools.build.lib.buildtool.SkyframeBuilder.buildArtifacts(SkyframeBuilder.java:146)
at com.google.devtools.build.lib.buildtool.ExecutionTool.executeBuild(ExecutionTool.java:467)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:221)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:494)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:462)
at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:103)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:610)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:234)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:556)
at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:620)
at io.grpc.Context$1.run(Context.java:566)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.