bazelbuild / bazelbuild/bazel

[8.0.1] External symlinks missing in execroot

Open
#28,915 3 comments 0 reactions 0 assignees View on GitHub
team-ExternalDeps type: bug under investigation
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

### Description of the bug:

A single engineer repeatedly hits build errors like
```
ERROR: <...>/BUILD.bazel:5305:12: output 'hdrs/thirdparty/build/jet/table.h' is a dangling symbolic link
ERROR: <...>/BUILD.bazel:5305:12: Symlink hdrs/thirdparty/build/jet/table.h failed: not all outputs were created or valid
ERROR: <...>/BUILD.bazel:5305:12: output 'hdrs/cyaml/cyaml.h' is a dangling symbolic link
ERROR: <...>/BUILD.bazel:5305:12: Symlink hdrs/cyaml/cyaml.h failed: not all outputs were created or valid
```
which we traced back to a missing `//execroot/_main/external/` directory. The rest of the execroot seems to have been populated correctly. The `//external/ ` directory is also present and populated correctly, suggesting the external bzlmod repos have been successfully fetched and built.

No reliable repro steps have been found and only 1 engineer is observing it. The problem has been observed in a totally clean workspace, with the disk cache deleted and having run `bazel clean --expunge`. It has also been observed when running the build with all user environment disabled (running under `env -i`) and no user config (`HOME` pointing to an empty directory and `--bazelrc=/dev/null` passed).

It appears to be intermittent but will get stuck in a failing state until something is changed about the build, like making a source code change or passing in a new config flag. But only passing `--noexperimental_merged_skyframe_analysis_execution` has been found to reliably fix the build.

I appreciate there may not be much you can do in this case, in the absence of repro steps, but would appreciate pointers to what we should investigate to diagnose the issue.

### Which category does this issue belong to?

_No response_

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Unfortunately, I don't have repro steps. Because we only hit this for a single user, and even then it was intermittent with no set of steps found to reliably reproduce the issue.

### Which operating system are you running Bazel on?

RHEL 8.10

### What is the output of `bazel info release`?

release 8.0.0- (@non-git)

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

Fetched and unpacked https://github.com/bazelbuild/bazel/releases/download/8.0.1/bazel-8.0.1-dist.zip
Built using
1) env SOURCE_DATE_EPOCH=1 EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh
2) output/bazel build //scripts:bazel-complete.bash //src/main/tools:linux-sandbox //src/tools/execlog:parser_deploy.jar //src/tools/execlog:converter_deploy.jar

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text

```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

_No response_

### Have you found anything relevant by searching the web?

[https://github.com/bazelbuild/bazel/issues/24619 has similar symptoms and workaround, but seemingly a different trigger

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by comparing the missing //execroot/_main/external/ directory with the populated //external/ directory during a failing build. Use the reported --noexperimental_merged_skyframe_analysis_execution workaround as a comparison point; done means isolating a reproducible trigger or a clear diagnosis for the missing external symlinks.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.