[7.6.1] `build --features=thin_lto` may fail with `error reading imports file .o.imports: Missing digest:`
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
Hello recently I upgraded Bazel from 6.5.0 to 7.6.1 and ran into an error about remote cache, I tried `--experimental_remote_cache_eviction_retries=5` but it doesn't help.
``` log
LTO Backend Compile bazel-out/arm-opt/bin/path/to/libbar.so.lto/bazel-out/arm-opt/bin/path/to/_objs/another/another.pic.o failed: \
error reading imports file /workdir/workspace/repo/.cache/execroot/my_workspace/bazel-out/arm-opt/bin/path/to/libbar.so.lto/bazel-out/arm-opt/bin/path/to/_objs/another/another.pic.o.imports: \
Missing digest: HASH/LENGTH for bazel-out/arm-opt/bin/path/to/libbar.so.lto/bazel-out/arm-opt/bin/path/to/_objs/another/another.pic.o
```
### Details
- `--remote_download_outputs` is the default value `toplevel`
- and I also enabled `--experimental_remote_cache_eviction_retries=5`. So I once thought Bazel would auto retry for several times.
- `--remote_local_fallback=true`
- `--incompatible_allow_tags_propagation` is the default value `true`
``` bazel
cc_binary(
name = "bar",
linkshared = True,
# I want to use "no-cache" to disable uploading libbar.so (which is too big: about 1.5GB),
# however it seems `LTO indexing` inherits all the tags (though `LTO Backend Compile doesn't inherit them)
tags = ["no-cache", "no-remote"],
deps = [...],
)
```
And the command line was `bazel build --remote_cache=http://my_cache //path/to:some_pkg`
But the building log has **nothing like** `ERROR: Build did NOT complete successfully\nFound transient remote cache error, retrying the build...`
After reading source code of version 7.6.1, I think it's a bug of Bazel:
- `experimental_remote_cache_eviction_retries` is only checked by
- `RemoteSpawnRunner::exec` -> `execLocallyAndUploadOrFail` -> `handleError`
- `LocalSpawnRunner::exec` -> `prefetchInputsAndWait` -> `prefetchInputs`
- `BlazeCommandDispatcher::exec` -> `execExclusively`
- but, **`error reading imports file`** is in `LtoBackendAction::discoverInputs`,
- called by `ActionExecutionFunction::checkCacheAndExecuteIfNeeded`
- called by `ActionExecutionFunction::compute`
- called by `AbstractParallelEvaluator::run`
- (there're too many `.run()`, not surce where to call it)
- Then the `LtoBackendAction::discoverInputs` is not protected by `remoteRetryOnTransientCacheError`
### Which category does this issue belong to?
Remote Execution
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
_No response_
### Which operating system are you running Bazel on?
Ubuntu 20.04, x86_64
### What is the output of `bazel info release`?
release 7.6.1
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
(A private repo)
```
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
_No response_
### Have you found anything relevant by searching the web?
Not found any `LTO`+`remote-cache` issue.
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start by tracing LtoBackendAction.discoverInputs through ActionExecutionFunction.checkCacheAndExecuteIfNeeded and the evaluator path, alongside the retry paths named in the report. Reproduce with Bazel 7.6.1, thin LTO, remote caching, and the reported tags if possible. Done means the missing-digest failure and retry behavior are understood and covered by an appropriate regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- build-system, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100