bazelbuild / bazelbuild/bazel

[7.6.1] `build --features=thin_lto` may fail with `error reading imports file .o.imports: Missing digest:`

Open
#26,955 19 comments 0 reactions 0 assignees View on GitHub
help wanted P3 potential 7.x cherry-picks team-Remote-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

Hello recently I upgraded Bazel from 6.5.0 to 7.6.1 and ran into an error about remote cache, I tried `--experimental_remote_cache_eviction_retries=5` but it doesn't help.
``` log
LTO Backend Compile bazel-out/arm-opt/bin/path/to/libbar.so.lto/bazel-out/arm-opt/bin/path/to/_objs/another/another.pic.o failed: \
error reading imports file /workdir/workspace/repo/.cache/execroot/my_workspace/bazel-out/arm-opt/bin/path/to/libbar.so.lto/bazel-out/arm-opt/bin/path/to/_objs/another/another.pic.o.imports: \
Missing digest: HASH/LENGTH for bazel-out/arm-opt/bin/path/to/libbar.so.lto/bazel-out/arm-opt/bin/path/to/_objs/another/another.pic.o
```

### Details

- `--remote_download_outputs` is the default value `toplevel`
- and I also enabled `--experimental_remote_cache_eviction_retries=5`. So I once thought Bazel would auto retry for several times.
- `--remote_local_fallback=true`
- `--incompatible_allow_tags_propagation` is the default value `true`

``` bazel
cc_binary(
name = "bar",
linkshared = True,
# I want to use "no-cache" to disable uploading libbar.so (which is too big: about 1.5GB),
# however it seems `LTO indexing` inherits all the tags (though `LTO Backend Compile doesn't inherit them)
tags = ["no-cache", "no-remote"],
deps = [...],
)
```

And the command line was `bazel build --remote_cache=http://my_cache //path/to:some_pkg`

But the building log has **nothing like** `ERROR: Build did NOT complete successfully\nFound transient remote cache error, retrying the build...`

After reading source code of version 7.6.1, I think it's a bug of Bazel:
- `experimental_remote_cache_eviction_retries` is only checked by
- `RemoteSpawnRunner::exec` -> `execLocallyAndUploadOrFail` -> `handleError`
- `LocalSpawnRunner::exec` -> `prefetchInputsAndWait` -> `prefetchInputs`
- `BlazeCommandDispatcher::exec` -> `execExclusively`
- but, **`error reading imports file`** is in `LtoBackendAction::discoverInputs`,
- called by `ActionExecutionFunction::checkCacheAndExecuteIfNeeded`
- called by `ActionExecutionFunction::compute`
- called by `AbstractParallelEvaluator::run`
- (there're too many `.run()`, not surce where to call it)
- Then the `LtoBackendAction::discoverInputs` is not protected by `remoteRetryOnTransientCacheError`

### Which category does this issue belong to?

Remote Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

_No response_

### Which operating system are you running Bazel on?

Ubuntu 20.04, x86_64

### What is the output of `bazel info release`?

release 7.6.1

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text
(A private repo)
```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

_No response_

### Have you found anything relevant by searching the web?

Not found any `LTO`+`remote-cache` issue.

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing LtoBackendAction.discoverInputs through ActionExecutionFunction.checkCacheAndExecuteIfNeeded and the evaluator path, alongside the retry paths named in the report. Reproduce with Bazel 7.6.1, thin LTO, remote caching, and the reported tags if possible. Done means the missing-digest failure and retry behavior are understood and covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
build-system, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.