bazelbuild / bazelbuild/bazel

ctx.actions.symlink() are sent to RBE server as regular file, not symlinks

Open
#26,551 6 comments 0 reactions 0 assignees View on GitHub
P3 team-Remote-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 16h
Merged PRs (30d)
72

Description

### Description of the bug:

Let's say we have a symlink created by `ctx.actions.symlink(target_file=)`. When the symlink is used in a downstream action as an input, it is sent to the RBE server as a regular file, not a symlink.

### Which category does this issue belong to?

Remote Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

1. Set up RBE. To quickly reproduce the error, I used this quickstart guide: https://developers.google.com/remote-build-execution/docs/quickstart , and just cloned https://github.com/bazelbuild/bazel-toolchains directly. Then cherry-pick https://github.com/jacky8hyf/bazel-toolchains/commit/0f51617afff93b65dde9bc82d948f8bd61495c44 (Or alternatively, just clone my repository and checkout that 0f51617afff93b65dde9bc82d948f8bd61495c44).
2. Run the command (remote instance name redacted; I changed --remote_executor for my instance as well.)

```
bazelisk --bazelrc=bazelrc/latest.bazelrc build \
--config=remote \
--remote_instance_name=<...> \
--remote_executor=grpcs://us-west1-remotebuildexecution.googleapis.com \
//examples/remotebuildexecution/hello_world/cc/foo:gen
```

This fails with the following error:

```
INFO: Analyzed target //examples/remotebuildexecution/hello_world/cc/foo:gen (1 packages loaded, 2 targets configured).
ERROR: <...>/bazel-toolchains/examples/remotebuildexecution/hello_world/cc/foo/BUILD:7:8: Executing genrule //examples/remotebuildexecution/hello_world/cc/foo:gen failed: (Exit 127): bash failed: error executing Genrule command (from target //examples/remotebuildexecution/hello_world/cc/foo:gen) /bin/bash -c ... (remaining 1 argument skipped)
bazel-out/k8-opt-exec-ST-d293173514a2/bin/examples/remotebuildexecution/hello_world/cc/foo/hello_world.exe: error while loading shared libraries: libST-d293173514a2_examples_Sremotebuildexecution_Shello_Uworld_Scc_Slibsay_Uhello.so: cannot open shared object file: No such file or directory
Target //examples/remotebuildexecution/hello_world/cc/foo:gen failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 2.846s, Critical Path: 2.71s
INFO: 2 processes: 6 action cache hit, 2 internal.
ERROR: Build did NOT complete successfully
```

#### Analysis

I made the following changes in https://github.com/jacky8hyf/bazel-toolchains/commit/0f51617afff93b65dde9bc82d948f8bd61495c44 to reproduce the error:

1. Add `@platforms` in MODULE.bazel because `@rbe_default` needs it
2. Add `linkstatic=False` to ensure `hello_world` links to `say_hello` dynamically.
3. Use a `native_binary()` in **a different package at a different level** to wrap the `//examples/remotebuildexecution/hello_world/cc:hello_world` target. Here, I am using `//examples/remotebuildexecution/hello_world/cc/foo:hello_world`. Note that `native_binary()` is implemented with `ctx.actions.symlink()`: https://github.com/bazelbuild/bazel-skylib/blob/27d429d8d036af3d010be837cc5924de1ca8d163/rules/native_binary.bzl#L25
4. Use a `genrule()` that invokes the `native_binary()`.

The error is caused by the following:

1. `hello_world` binary has the following in `RUNPATH`:
```
0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN/../../../../_solib_k8/:$ORIGIN/hello_world.runfiles/_main/_solib_k8/]
```
This works when the `genrule` uses the `//examples/remotebuildexecution/hello_world/cc:hello_world` binary directly.
2. `native_binary()` creates a symlink to the binary. The symlink is at `bazel-out/k8-opt-exec-ST-d293173514a2/bin/examples/remotebuildexecution/hello_world/cc/foo/hello_world.exe`. There is no `hello_world.runfiles` next to it.
3. When building the `genrule()`, **even though `bazel-out/k8-opt-exec-ST-d293173514a2/bin/examples/remotebuildexecution/hello_world/cc/foo/hello_world.exe` is a symlink, it is sent as a regular file**. However, the `RUNPATH` would no longer work because there is no `bazel-out/k8-opt-exec-ST-d293173514a2/bin/examples/remotebuildexecution/hello_world/cc/foo/hello_world.runfiles`.

### Which operating system are you running Bazel on?

Linux

### What is the output of `bazel info release`?

8.3.1

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text

```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

_No response_

### Have you found anything relevant by searching the web?

I think this issue is quite similar to https://github.com/bazelbuild/bazel/issues/23620, except that in https://github.com/bazelbuild/bazel/issues/23620, the symlink is not "registered" to Bazel, but just exists in the filesystem.

Here, we are already "registering" this symlink to Bazel with `ctx.actions.symlink()` (note that Bazel created this symlink for us!), but it is still not recognized as a symlink in RBE. On the other hand, `ctx.actions.declare_symlink()`s are recognized as symlinks.

### Any other information, logs, or outputs that you want to share?

To workaround the issue, we could change `ctx.actions.symlink()` to `ctx.actions.declare_symlink()`. However, we can only work around it in our usages of `ctx.actions.symlink()`, but we can't work around it for `ctx.actions.symlink()` in external dependencies like `rules_cc` or `rules_rust`.

Here's our workaround: http://r.android.com/q/topic:%22b/430302030%22

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the bazel-toolchains example and the remote configuration described in the issue, then inspect ctx.actions.symlink(), ctx.actions.declare_symlink(), and the native_binary() rule in bazel-skylib. Verify how the downstream genrule input is transferred to RBE. Done means the symlink remains a symlink for the remote action and the example build succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
build-system, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.