[8.0.1] External symlinks missing in execroot
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 75
Description
### Description of the bug:
A single engineer repeatedly hits build errors like
```
ERROR: <...>/BUILD.bazel:5305:12: output 'hdrs/thirdparty/build/jet/table.h' is a dangling symbolic link
ERROR: <...>/BUILD.bazel:5305:12: Symlink hdrs/thirdparty/build/jet/table.h failed: not all outputs were created or valid
ERROR: <...>/BUILD.bazel:5305:12: output 'hdrs/cyaml/cyaml.h' is a dangling symbolic link
ERROR: <...>/BUILD.bazel:5305:12: Symlink hdrs/cyaml/cyaml.h failed: not all outputs were created or valid
```
which we traced back to a missing `//execroot/_main/external/` directory. The rest of the execroot seems to have been populated correctly. The `//external/ ` directory is also present and populated correctly, suggesting the external bzlmod repos have been successfully fetched and built.
No reliable repro steps have been found and only 1 engineer is observing it. The problem has been observed in a totally clean workspace, with the disk cache deleted and having run `bazel clean --expunge`. It has also been observed when running the build with all user environment disabled (running under `env -i`) and no user config (`HOME` pointing to an empty directory and `--bazelrc=/dev/null` passed).
It appears to be intermittent but will get stuck in a failing state until something is changed about the build, like making a source code change or passing in a new config flag. But only passing `--noexperimental_merged_skyframe_analysis_execution` has been found to reliably fix the build.
I appreciate there may not be much you can do in this case, in the absence of repro steps, but would appreciate pointers to what we should investigate to diagnose the issue.
### Which category does this issue belong to?
_No response_
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Unfortunately, I don't have repro steps. Because we only hit this for a single user, and even then it was intermittent with no set of steps found to reliably reproduce the issue.
### Which operating system are you running Bazel on?
RHEL 8.10
### What is the output of `bazel info release`?
release 8.0.0- (@non-git)
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
Fetched and unpacked https://github.com/bazelbuild/bazel/releases/download/8.0.1/bazel-8.0.1-dist.zip
Built using
1) env SOURCE_DATE_EPOCH=1 EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh
2) output/bazel build //scripts:bazel-complete.bash //src/main/tools:linux-sandbox //src/tools/execlog:parser_deploy.jar //src/tools/execlog:converter_deploy.jar
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
```
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
_No response_
### Have you found anything relevant by searching the web?
[https://github.com/bazelbuild/bazel/issues/24619 has similar symptoms and workaround, but seemingly a different trigger
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start by comparing the missing //execroot/_main/external/ directory with the populated //external/ directory during a failing build. Use the reported --noexperimental_merged_skyframe_analysis_execution workaround as a comparison point; done means isolating a reproducible trigger or a clear diagnosis for the missing external symlinks.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100