bazelbuild / bazelbuild/bazel

Identical exec configurations get different output directory ST- hashes depending on top-level target config

Open
#28,989 5 comments 0 reactions 0 assignees View on GitHub
P3 team-Configurability type: support / not a bug (process)
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

## Title

Identical exec configurations get different output directory ST- hashes depending on top-level target config

## Description

Two exec-configuration targets with **byte-for-byte identical `BuildConfigurationValue` content** (same full config ID) receive different `-ST-` output directory suffixes depending on the top-level target configuration (e.g., `fastbuild` vs `opt`). This prevents cache reuse (both local and remote) for all exec-config actions when switching between compilation modes.

### Environment

- Bazel 9.0.1
- macOS (aarch64), but the issue is platform-independent
- `--experimental_output_directory_naming_scheme=DIFF_AGAINST_DYNAMIC_BASELINE` (default)
- `--experimental_exec_configuration_distinguisher=OFF` (default)

### Setup

We use `rules_rust` with Starlark build settings (`@rules_rust//rust/settings:lto`, `extra_rustc_flag`, `incremental`, `pipelined_compilation`, etc.) configured via `.bazelrc`:

```
# Default build (fastbuild):
# No extra Starlark settings

# Optimized build (--config=opt):
common:opt --compilation_mode=opt
common:opt --@rules_rust//rust/settings:lto=thin
common:opt --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
common:opt --@rules_rust//rust/settings:extra_rustc_flag=--codegen=codegen-units=1
# ... more opt flags
```

We have a Starlark rule transition (`exec_settings_trim.bzl`) that resets all these settings to defaults for exec-configuration targets (proc macros, build scripts), since they are irrelevant for tools. This produces identical exec configurations regardless of the top-level build mode.

### Reproduction

```bash
# Build in fastbuild mode
bazel build //my:target --config=nobes

# Check the output path for an exec-config crate (e.g., reqwest, used by a proc macro)
bazel aquery 'mnemonic("Rustc", deps(//my:target))' --config=nobes 2>/dev/null \
| grep 'Outputs:.*reqwest'
# Output: bazel-out/aarch64-darwin-opt-ST-3e9461dcbca5/bin/.../libreqwest.rlib

# Build in opt mode
bazel build //my:target --config=nobes --config=opt

# Check the same exec-config crate
bazel aquery 'mnemonic("Rustc", deps(//my:target))' --config=nobes --config=opt 2>/dev/null \
| grep 'Outputs:.*reqwest'
# Output: bazel-out/aarch64-darwin-opt-ST-b4c5f3c7db7f/bin/.../libreqwest.rlib
# ^^^^^^^^^^^^
# DIFFERENT ST- hash!
```

### Verification that config content is identical

```bash
# The full config ID is the same in both builds:
# 17eb2ad9cf6c6e4796359b4fd7c1a9bccd6355b89da4b1c627db30ca3c9d2672

# Dump the config after each build and diff:
bazel build //my:target --config=nobes
bazel config 17eb2ad9... > /tmp/config-fastbuild.txt

bazel build //my:target --config=nobes --config=opt
bazel config 17eb2ad9... > /tmp/config-opt.txt

diff /tmp/config-fastbuild.txt /tmp/config-opt.txt
# Empty diff — byte-for-byte identical config content
```

The `affected by starlark transition` field is also empty `[]` in both cases.

### Root cause analysis

The ST- hash is computed by `OutputPathMnemonicComputer` as a diff between the configuration's options and a **dynamic baseline**. The dynamic baseline is derived from the top-level command-line configuration after applying the native exec transition.

The native exec transition does **not** reset user-defined Starlark build settings. So when `--config=opt` sets `@rules_rust//rust/settings:lto=thin`, this leaks into the exec baseline. The baseline for `fastbuild` does not have `lto=thin`.

Our Starlark transition resets `lto` to `"unspecified"` in the downstream config. But:

- `diff(config, fastbuild_baseline)` — the baseline has `lto=unspecified`, so no diff on `lto`
- `diff(config, opt_baseline)` — the baseline has `lto=thin`, so there IS a diff on `lto`

Different diffs → different ST- hashes → different output directories → cache miss.

This also explains why `fastbuild` and `dbg` share the same ST- hash (neither sets Starlark flags, so their baselines are identical), while `opt` diverges.

### What we tried

| Approach | Result |
|---|---|
| `--incompatible_exclude_starlark_flags_from_exec_config` | Stabilizes the native exec baseline config ID, but downstream Starlark transitions still get different ST- hashes |
| Custom exec transition via `--experimental_exec_config` | User-defined `transition()` cannot read/write `--experimental_*` / `--incompatible_*` flags, so it cannot replicate the built-in exec transition |
| `--experimental_output_paths=strip` | Causes correctness bugs (conflates LTO and non-LTO artifacts) |
| All three `--experimental_output_directory_naming_scheme` values | None fix it |

### Impact

Switching between `fastbuild`/`dbg` and `opt` causes a full rebuild of **all** exec-configuration targets and their transitive dependencies (~3,200 actions in our workspace), even though the rustc commands are byte-for-byte identical. Neither local disk cache nor remote cache can be shared across these modes for exec-config actions.

### Expected behavior

Two configurations with identical `BuildConfigurationValue` content (same full config ID) should always receive the same output directory path, regardless of the transition path that produced them.

### Suggested fixes (any one would resolve this)

1. **Fix the baseline computation**: When computing the ST- hash for a config, use the config's own content to derive a canonical baseline (e.g., the exec baseline with Starlark flags stripped), rather than a baseline derived from the top-level target config. This would ensure identical configs always diff against the same baseline.

2. **Make `--incompatible_exclude_starlark_flags_from_exec_config` propagate to Starlark transition baselines**: Currently it only affects the native exec transition output. Downstream Starlark transitions still compute their ST- hash against a baseline that includes the leaked Starlark flags.

3. **Allow `--experimental_exec_config` to reference user .bzl transitions**: Currently, user-defined `transition()` cannot read/write `--experimental_*`/`--incompatible_*` flags, making it impossible to create a custom exec transition that extends the built-in one. Relaxing this restriction (or providing an API to compose with the built-in exec transition) would let rule authors fix this in their own rulesets.

### Related issues

- #14023 — Avoid different transition output directories when build settings have the same values (closed, partially addressed by `diff_against_baseline`)
- #12171 — Output directory hash differs if the same build settings are applied via transition vs. command line
- #14239 — Action conflicts due to differing `affected by starlark transition` value

Contributor guide

Open the contributing guide

Research direction

Start with OutputPathMnemonicComputer and the exec_settings_trim.bzl transition, then reproduce the differing ST- hashes using the provided bazel build and aquery commands. Compare the baseline and downstream configuration data described in the issue; done means byte-for-byte identical BuildConfigurationValue content produces the same output directory hash across fastbuild and opt.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.