bazelbuild / bazelbuild/bazel

Fully remote build: host platform leaks into configuration, trashing the caching

Open
#26,357 7 comments 0 reactions 1 assignee Claimed by @tjgq View on GitHub
team-Configurability type: bug untriaged
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

### Description of the bug:

I'm trying to set up a fully remote build using bazel with RBE (I don't think it matters, but I'm using BuildBarn). By fully remote build I understand the setup where all actions run on RBE, and the only thing running locally is bazel itself. To keep things simple, currently all RBE workers are running on Linux amd64 and the only supported target is also Linux amd64. However, clients can be both on Linux and Macs.

So, I've defined my custom platform, essentially just saying it's Linux x86_64 + setting some exec_properties:
```
platform(
name = "remote-amd64",
constraint_values = [
"@platforms//cpu:x86_64",
"@platforms//os:linux",
],
exec_properties = {
"OSFamily": "linux",
},
)
```

Then I've added `--platforms` and `--extra_execution_platforms` to my workspace `.bazelrc`:
```
build:remote --platforms=//build/rules:remote-amd64
build:remote --extra_execution_platforms=//build/rules:remote-amd64
```

It works, but if I run bazel from a Mac, all the outputs are created under `bazel-out/darwin_arm64-dbg`. That's annoying, since the actual binaries are compiled for Linux amd64. Moreover, it trashes the cashing with generated inputs, since they now have different paths if built from Linux/Mac resp.

Ok, I've googled up the `--experimental_platform_in_output_dir` switch and it makes the output path consistent indeed.

But today I found that there is still no cache sharing between Linux and Mac clients. Started digging it up and found that this dreaded `darwin_arm64` thing, even though no longer in the out path, still leaks into the configuration, making it a cache miss:
```
$ bazel aquery 'mnemonic("Genrule", //test:test_file)' --output=textproto | grep -A 5 "configuration {"
configuration {
id: 1
mnemonic: "remote-amd64-dbg"
platform_name: "darwin_arm64"
checksum: "fc6ea996c624cdfcf4b382607745a24d429545b45e5d49aa4f2a56cd08fe2dd3"
}
```

WTH? Is it a bug? Am I missing something in my configuration?

I've googled suggestions to also set `--host_platform=//build/rules:remote-amd64`, I've tried that too, even though I don't think that's correct according to the platforms documentation, but to no avail.

I've put a small sample repo here: https://github.com/yanok/full-remote-build-host-platform
You won't be able to build, our build cluster is on the intranet, sorry. But you should be able to run `bazel aquery` on it.

I'm more than ready to send a PR for this, but could someone from bazel team first confirm that:
1. "Fully remote" build is a supported configuration
2. There is no way to fix it with a command line switch
3. It is indeed a bug and not a feature

### Which category does this issue belong to?

_No response_

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

_No response_

### Which operating system are you running Bazel on?

MacOS

### What is the output of `bazel info release`?

release 8.2.1

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text

```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.