Non-hermeticity of virtual headers without sandboxing leads to cache poisoning
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
I've noticed that virtual include directories (e.g. when specifying the `strip_include_prefix` attribute of the `cc_library` rule) where headers are copied/linked are not re-created when the set of declared `hdrs` changes. This means meas that old headers are still present in a created directory, even if they are removed from the `hdrs` list if a `bazel clean` is not performed.
This non-hermeticity can easily lead to cache poisoning when sandboxing is disabled (primarily under Windows or by using `--spawn_strategy=local`) and, thus, subsequent false failures of the dependency inclusion checker until the remote/disk cache is pruned or disabled.
Let me give an example. I'll use an auto-detected toolchain on Linux with sandboxing disabled but the issue was found on Windows with a custom, hermetic, LLVM-based toolchain.
So, consider a project with a `cc_library` that depends on some other `cc_library` that provides some headers with `strip_include_prefix`. At some point, a user decides to switch the source of one of the headers to a different `cc_library`. As such, it removes that header from `hdrs` of the original dependency and adds a new dependency to `deps` of the top-level `cc_library` target. Assuming incremental builds, if no clean is performed after the removal of the header from `hdrs` and disk/remote cache was enabled, Bazel will fail on the dependency inclusion checker. This is expected, since the old header is still linked in virtual headers directory of the original `cc_library` dependency but there is not dependency on it declared in a `BUILD` file. Yet, that failure will be cached (I assume due to the dependency file being produced since the underlying compilation itself will be successful). Now, with no remote/disk cache, a simple clean and rebuild would be enough make the build work, as the virtual headers directory of the original `cc_library` dependency would be removed and created without that old headers. Unfortunately, with remote/disk cache the issue will persist even after clean since Bazel will detect that inputs did not change (which is technically true) and return the same, cached, error message from the dependency checker (even though, the actual contents of the virtual header directories are different). At this point the only way to make build pass (as would be expected) is to either disable of prune the remote/disk cache.
Reproduction example:
[non_hermetic_virtual_includes_cache_poisoning.zip](https://github.com/user-attachments/files/25090067/non_hermetic_virtual_includes_cache_poisoning.zip)
### Which category does this issue belong to?
C++ Rules
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Assuming using the reproduction example I provided above.
Starting situation:
```python
# BUILD
cc_library(
name = "f",
srcs = ["f.c"],
deps = [
"//dep1:dep1_h",
],
)
```
```python
# dep1/BUILD
cc_library(
name = "dep1_h",
hdrs = [
"some_inc.h",
"other_inc.h",
],
strip_include_prefix = ".",
visibility = ["//visibility:public"],
)
```
```c
// f.c
#include "some_inc.h"
#include "other_inc.h"
//...
```
Initial build from a clean environment, using disk cache:
```
$ bazel build //:f --disk_cache=../disk_cache_test
...
INFO: 6 processes: 3 internal, 3 local.
INFO: Build completed successfully, 6 total actions
$ ls -1 bazel-bin/dep1/_virtual_includes/dep1_h/
other_inc.h
some_inc.h
```
Now, I decide I want a different library to provide the `some_inc.h` header:
```python
# BUILD
cc_library(
name = "f",
srcs = ["f.c"],
deps = [
"//dep1:dep1_h",
"//dep2:dep2_h",
],
)
```
```python
# dep1/BUILD
cc_library(
name = "dep1_h",
hdrs = [
"other_inc.h",
],
strip_include_prefix = ".",
visibility = ["//visibility:public"],
)
```
```python
# dep2/BUILD
cc_library(
name = "dep2_h",
hdrs = [
"some_inc.h",
],
strip_include_prefix = ".",
visibility = ["//visibility:public"],
)
```
I perform an incremental build:
```
$ bazel build //:f --disk_cache=../disk_cache_test
...
ERROR: /mnt/c/Users/bqwufr/Aptiv/virtual_includes_non_hermetic_bazel8_generic_reproduction/BUILD:1:11: Compiling f.c failed: undeclared inclusion(s) in rule '//:f':
this rule is missing dependency declarations for the following files included by 'f.c':
'bazel-out/k8-fastbuild/bin/dep1/_virtual_includes/dep1_h/some_inc.h'
Target //:f failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.847s, Critical Path: 0.15s
INFO: 3 processes: 1 action cache hit, 3 internal.
ERROR: Build did NOT complete successfully
```
The build fails, since `some_inc.h` is still present in the virtual includes directory of `//dep1:dep1_h`:
```
$ ls -1 bazel-bin/dep1/_virtual_includes/dep1_h/
other_inc.h
some_inc.h
$ ls -1 bazel-bin/dep2/_virtual_includes/dep2_h/
some_inc.h
```
Now, a clean will make sure that the virtual includes directory of `//dep1:dep1_h` is removed and created again, now with correct headers, so that a clean rebuild should be successful:
```
$ bazel clean
...
$ bazel build //:f --disk_cache=../disk_cache_test
...
ERROR: /mnt/c/Users/bqwufr/Aptiv/virtual_includes_non_hermetic_bazel8_generic_reproduction/BUILD:1:11: Compiling f.c failed: undeclared inclusion(s) in rule '//:f':
this rule is missing dependency declarations for the following files included by 'f.c':
'bazel-out/k8-fastbuild/bin/dep1/_virtual_includes/dep1_h/some_inc.h'
Target //:f failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.237s, Critical Path: 0.02s
INFO: 4 processes: 4 internal.
ERROR: Build did NOT complete successfully
```
But it's not since the fail is cached (due to non-hermetic behavior of virtual includes) and inputs (technically) did not change. Yet, once the remote/disk cache is disabled (or pruned), the build will be successful, as expected:
```
$ bazel build //:f
...
INFO: 4 processes: 2 action cache hit, 1 internal, 3 local.
INFO: Build completed successfully, 4 total actions
```
### Which operating system are you running Bazel on?
Applicable to all OSes
### What is the output of `bazel info release`?
release 7.1.0 - release 8.5.1
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
n/a
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
n/a
```
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
n/a
### Have you found anything relevant by searching the web?
Nothing specifically related to this particular use case.
### Any other information, logs, or outputs that you want to share?
n/a
Contributor guide
Research direction
Start with the attached reproduction and the BUILD, dep1/BUILD, dep2/BUILD, and f.c files. Run the incremental Bazel builds with --disk_cache, inspect the _virtual_includes directories, then compare clean and cache-disabled builds. Done means changing hdrs removes the obsolete virtual header and no stale dependency-checker failure is returned from the cache.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100