bazelbuild / bazelbuild/bazel-skylib
Backticks in analysis test string literals are evaluated as shell script
- Dominant language
- Starlark
- Stars
- 444
- Forks
- 202
- PR merge metrics
- No merged PRs in 30d
Description
I have a minimal reproduction repo here: https://github.com/dgp1130/skylib-test-eval. (I'm using Linux. Windows may or may not reproduce.)
TL;DR: Using `analysistest` to assert on an error message containing backticks will evaluate the text in those backticks in the shell.
An innocuous assertion like:
```starlark
asserts.expect_failure(env, "Oh noes! Value `foo` should have been `bar`!")
```
Will output failure messages like:
```
==================== Test output for //:failure_testing_test:
/home/douglasparker/.cache/bazel/_bazel_douglasparker/f8daf9e1b24fe39f31e93c50d2583d6b/sandbox/linux-sandbox/9/execroot/__main__/bazel-out/k8-fastbuild/bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: foo: command not found
/home/douglasparker/.cache/bazel/_bazel_douglasparker/f8daf9e1b24fe39f31e93c50d2583d6b/sandbox/linux-sandbox/9/execroot/__main__/bazel-out/k8-fastbuild/bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: bar: command not found
/home/douglasparker/.cache/bazel/_bazel_douglasparker/f8daf9e1b24fe39f31e93c50d2583d6b/sandbox/linux-sandbox/9/execroot/__main__/bazel-out/k8-fastbuild/bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: bar: command not found
/home/douglasparker/.cache/bazel/_bazel_douglasparker/f8daf9e1b24fe39f31e93c50d2583d6b/sandbox/linux-sandbox/9/execroot/__main__/bazel-out/k8-fastbuild/bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: foo: command not found
In test _failure_testing_test_impl from //:build_defs_test.bzl: Expected errors to contain 'Oh noes! Value should have been !' but did not. Actual errors:
Traceback (most recent call last):
File "/home/douglasparker/Source/bazel-test-eval/BUILD", line 3
_my_rule(name = 'this_should_fail')
File "/home/douglasparker/Source/bazel-test-eval/build_defs_test.bzl", line 15, in _my_rule_impl
fails_with_backtick_in_message()
File "/home/douglasparker/Source/bazel-test-eval/build_defs.bzl", line 2, in fails_with_backtick_in_message
fail(<1 more arguments>)
Oh noes! Value should have been !
```
It appears that `foo` and `bar` are being treated as a subshell and evaluated inline. I believe the evaluation is coming from [here](https://github.com/bazelbuild/bazel-skylib/blob/560d7b2359aecb066d81041cb532b82d7354561b/toolchains/unittest/BUILD#L35). Running one of the generated shell scripts reproduces the problem:
```
$ bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh
bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: foo: command not found
bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: bar: command not found
bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: bar: command not found
bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: foo: command not found
In test _failure_testing_test_impl from //:build_defs_test.bzl: Expected errors to contain 'Oh noes! Value should have been !' but did not. Actual errors:
Traceback (most recent call last):
File "/home/douglasparker/Source/bazel-test-eval/BUILD", line 3
_my_rule(name = 'this_should_fail')
File "/home/douglasparker/Source/bazel-test-eval/build_defs_test.bzl", line 15, in _my_rule_impl
fails_with_backtick_in_message()
File "/home/douglasparker/Source/bazel-test-eval/build_defs.bzl", line 2, in fails_with_backtick_in_message
fail(<1 more arguments>)
Oh noes! Value should have been !
$ cat bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh
cat << EOF
In test _failure_testing_test_impl from //:build_defs_test.bzl: Expected errors to contain 'Oh noes! Value `foo` should have been `bar`!' but did not. Actual errors:
Traceback (most recent call last):
File "/home/douglasparker/Source/bazel-test-eval/BUILD", line 3
_my_rule(name = 'this_should_fail')
File "/home/douglasparker/Source/bazel-test-eval/build_defs_test.bzl", line 15, in _my_rule_impl
fails_with_backtick_in_message()
File "/home/douglasparker/Source/bazel-test-eval/build_defs.bzl", line 2, in fails_with_backtick_in_message
fail(<1 more arguments>)
Oh noes! Value `bar` should have been `foo`!
EOF
exit 1
```
Contributor guide
Research direction
Start with toolchains/unittest/BUILD around the linked line 35 and inspect the generated failure_testing_test.sh shown in the report. Run the minimal reproduction or generated script on Linux, then verify that an analysistest assertion containing backticks preserves them literally and reports the expected message without shell errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- shell
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100