ctx.actions.symlink() are sent to RBE server as regular file, not symlinks
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
Let's say we have a symlink created by `ctx.actions.symlink(target_file=)`. When the symlink is used in a downstream action as an input, it is sent to the RBE server as a regular file, not a symlink.
### Which category does this issue belong to?
Remote Execution
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
1. Set up RBE. To quickly reproduce the error, I used this quickstart guide: https://developers.google.com/remote-build-execution/docs/quickstart , and just cloned https://github.com/bazelbuild/bazel-toolchains directly. Then cherry-pick https://github.com/jacky8hyf/bazel-toolchains/commit/0f51617afff93b65dde9bc82d948f8bd61495c44 (Or alternatively, just clone my repository and checkout that 0f51617afff93b65dde9bc82d948f8bd61495c44).
2. Run the command (remote instance name redacted; I changed --remote_executor for my instance as well.)
```
bazelisk --bazelrc=bazelrc/latest.bazelrc build \
--config=remote \
--remote_instance_name=<...> \
--remote_executor=grpcs://us-west1-remotebuildexecution.googleapis.com \
//examples/remotebuildexecution/hello_world/cc/foo:gen
```
This fails with the following error:
```
INFO: Analyzed target //examples/remotebuildexecution/hello_world/cc/foo:gen (1 packages loaded, 2 targets configured).
ERROR: <...>/bazel-toolchains/examples/remotebuildexecution/hello_world/cc/foo/BUILD:7:8: Executing genrule //examples/remotebuildexecution/hello_world/cc/foo:gen failed: (Exit 127): bash failed: error executing Genrule command (from target //examples/remotebuildexecution/hello_world/cc/foo:gen) /bin/bash -c ... (remaining 1 argument skipped)
bazel-out/k8-opt-exec-ST-d293173514a2/bin/examples/remotebuildexecution/hello_world/cc/foo/hello_world.exe: error while loading shared libraries: libST-d293173514a2_examples_Sremotebuildexecution_Shello_Uworld_Scc_Slibsay_Uhello.so: cannot open shared object file: No such file or directory
Target //examples/remotebuildexecution/hello_world/cc/foo:gen failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 2.846s, Critical Path: 2.71s
INFO: 2 processes: 6 action cache hit, 2 internal.
ERROR: Build did NOT complete successfully
```
#### Analysis
I made the following changes in https://github.com/jacky8hyf/bazel-toolchains/commit/0f51617afff93b65dde9bc82d948f8bd61495c44 to reproduce the error:
1. Add `@platforms` in MODULE.bazel because `@rbe_default` needs it
2. Add `linkstatic=False` to ensure `hello_world` links to `say_hello` dynamically.
3. Use a `native_binary()` in **a different package at a different level** to wrap the `//examples/remotebuildexecution/hello_world/cc:hello_world` target. Here, I am using `//examples/remotebuildexecution/hello_world/cc/foo:hello_world`. Note that `native_binary()` is implemented with `ctx.actions.symlink()`: https://github.com/bazelbuild/bazel-skylib/blob/27d429d8d036af3d010be837cc5924de1ca8d163/rules/native_binary.bzl#L25
4. Use a `genrule()` that invokes the `native_binary()`.
The error is caused by the following:
1. `hello_world` binary has the following in `RUNPATH`:
```
0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN/../../../../_solib_k8/:$ORIGIN/hello_world.runfiles/_main/_solib_k8/]
```
This works when the `genrule` uses the `//examples/remotebuildexecution/hello_world/cc:hello_world` binary directly.
2. `native_binary()` creates a symlink to the binary. The symlink is at `bazel-out/k8-opt-exec-ST-d293173514a2/bin/examples/remotebuildexecution/hello_world/cc/foo/hello_world.exe`. There is no `hello_world.runfiles` next to it.
3. When building the `genrule()`, **even though `bazel-out/k8-opt-exec-ST-d293173514a2/bin/examples/remotebuildexecution/hello_world/cc/foo/hello_world.exe` is a symlink, it is sent as a regular file**. However, the `RUNPATH` would no longer work because there is no `bazel-out/k8-opt-exec-ST-d293173514a2/bin/examples/remotebuildexecution/hello_world/cc/foo/hello_world.runfiles`.
### Which operating system are you running Bazel on?
Linux
### What is the output of `bazel info release`?
8.3.1
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
```
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
_No response_
### Have you found anything relevant by searching the web?
I think this issue is quite similar to https://github.com/bazelbuild/bazel/issues/23620, except that in https://github.com/bazelbuild/bazel/issues/23620, the symlink is not "registered" to Bazel, but just exists in the filesystem.
Here, we are already "registering" this symlink to Bazel with `ctx.actions.symlink()` (note that Bazel created this symlink for us!), but it is still not recognized as a symlink in RBE. On the other hand, `ctx.actions.declare_symlink()`s are recognized as symlinks.
### Any other information, logs, or outputs that you want to share?
To workaround the issue, we could change `ctx.actions.symlink()` to `ctx.actions.declare_symlink()`. However, we can only work around it in our usages of `ctx.actions.symlink()`, but we can't work around it for `ctx.actions.symlink()` in external dependencies like `rules_cc` or `rules_rust`.
Here's our workaround: http://r.android.com/q/topic:%22b/430302030%22
Contributor guide
Research direction
Reproduce the failure with the bazel-toolchains example and the remote configuration described in the issue, then inspect ctx.actions.symlink(), ctx.actions.declare_symlink(), and the native_binary() rule in bazel-skylib. Verify how the downstream genrule input is transferred to RBE. Done means the symlink remains a symlink for the remote action and the example build succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux
- Domain
- build-system, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100