Runfile collisions across transitions
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
When building up runfiles there can be cases where the 'short_path' is identical
yet the actual file is different due to transitions. So when a rule is building up runfiles using ctx.runfiles
only one of the files that collides will be present in the runfiles.
The collision can be silent, giving no errors or indications it has occurred.
The attached zip contains a project workspace that recreates an issue with runfiles collisions when there are transitions involved.
For example:
```
DEBUG: /Users/****/repos/overwrote_runfiles/rules.bzl:92:18: File: short_path:hello_world.h path:bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/hello_world.h
DEBUG: /Users/****/repos/overwrote_runfiles/rules.bzl:92:18: File: short_path:hello_world.h path:bazel-out/darwin_arm64-fastbuild-ST-3af09e09b98c/bin/hello_world.h
```
This workspace is likely more complex than needs to be but it approximates a real world scenario that
exhibited these collisions.
The basics of it are there is a reporting rule that has dependencies on test rules. These test rules
run binaries but each test uses different transitions, one is the 'exec' and another a custom transition.
So the outputs of the binary are dependant on the incoming transition.
An aspect collects the source from the test targets. The reporting rule in this example does not do
anything with the sources, but in our real implementation the reporting gear uses sources,
objects and test outputs to create a global report about all the tests.
The issue with the collisions causes the following to happen when runfiles are created:
1. Silent collisions: Last file in the set wins
2. Symlink 'overwrote runfile': When creating runfiles with 'symlinks' then the collisions are logged as a error but do not fail. See https://github.com/bazelbuild/bazel/issues/4965
For our implementation we were able to squelch the error by filtering before creating runfiles, detecting collisions, and
only using the last File in the runfiles. This only works because the files that are produced as of the transitions
happen to be identical. This may not be true for other situations as the transition might cause differences in the
files created.
To recreate the issue
```
bazel run -s //:report
```
Debug statements show the File objects that ultimately end up as runfiles to the reporting rule.
```
DEBUG: /Users/****/repos/overwrote_runfiles/rules.bzl:92:18: File: short_path:hello_world.h path:bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/hello_world.h
DEBUG: /Users/****/repos/overwrote_runfiles/rules.bzl:92:18: File: short_path:hello_world.h path:bazel-out/darwin_arm64-fastbuild-ST-3af09e09b98c/bin/hello_world.h
```
If symlinks are used in the runfiles, the you will get the error that is not an error. See https://github.com/bazelbuild/bazel/issues/4965
`ERROR: /Users/****/repos/overwrote_runfiles/BUILD.bazel:41:7: overwrote runfile hello_world.h, was symlink to bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/hello_world.h, now symlink to bazel-out/darwin_arm64-fastbuild-ST-3af09e09b98c/bin/hello_world.h`
And in the action itself we can see only the last file added to the set of runfiles is symlinked into the actions space.
```
Running Report
total 0
lrwxr-xr-x 1 **** staff 151 29 Sep 09:47 hello_world.h -> /Volumes/BazelOut/_bazel_****/42a766200e36a3e8369aec1f430cf379/execroot/__main__/bazel-out/darwin_arm64-fastbuild-ST-3af09e09b98c/bin/hello_world.h
```
In my specific scenario, I actually don't really need the runfiles, as I have all the paths to the actual outputs, but I do need the actions to produce the outputs based on the 'test' dependencies, so the report
can access them.
Ultimately when having runfiles there should be no way to have collisions across transitions, however I do not see how that can be possible with the way runfiles are implemented today.
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Using the attached workspace zip file
```
bazel run -s //:report
```
### Which operating system are you running Bazel on?
MacOS
### What is the output of `bazel info release`?
_No response_
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
release 5.0.0
### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?
_No response_
### Have you found anything relevant by searching the web?
https://github.com/bazelbuild/bazel/issues/4965 is related but only for the error/waring Bazel emits.
### Any other information, logs, or outputs that you want to share?
[runfile_collisions_across_transisitions.zip](https://github.com/bazelbuild/bazel/files/9699617/runfile_collisions_across_transisitions.zip)
Contributor guide
Assessment
This issue has not been assessed yet.