bazelbuild / bazelbuild/bazel-skylib

Backticks in analysis test string literals are evaluated as shell script

Open
#255 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Starlark
Stars
444
Forks
202
PR merge metrics
No merged PRs in 30d

Description

I have a minimal reproduction repo here: https://github.com/dgp1130/skylib-test-eval. (I'm using Linux. Windows may or may not reproduce.)

TL;DR: Using `analysistest` to assert on an error message containing backticks will evaluate the text in those backticks in the shell.

An innocuous assertion like:

```starlark
asserts.expect_failure(env, "Oh noes! Value `foo` should have been `bar`!")
```

Will output failure messages like:

```
==================== Test output for //:failure_testing_test:
/home/douglasparker/.cache/bazel/_bazel_douglasparker/f8daf9e1b24fe39f31e93c50d2583d6b/sandbox/linux-sandbox/9/execroot/__main__/bazel-out/k8-fastbuild/bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: foo: command not found
/home/douglasparker/.cache/bazel/_bazel_douglasparker/f8daf9e1b24fe39f31e93c50d2583d6b/sandbox/linux-sandbox/9/execroot/__main__/bazel-out/k8-fastbuild/bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: bar: command not found
/home/douglasparker/.cache/bazel/_bazel_douglasparker/f8daf9e1b24fe39f31e93c50d2583d6b/sandbox/linux-sandbox/9/execroot/__main__/bazel-out/k8-fastbuild/bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: bar: command not found
/home/douglasparker/.cache/bazel/_bazel_douglasparker/f8daf9e1b24fe39f31e93c50d2583d6b/sandbox/linux-sandbox/9/execroot/__main__/bazel-out/k8-fastbuild/bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: foo: command not found
In test _failure_testing_test_impl from //:build_defs_test.bzl: Expected errors to contain 'Oh noes! Value should have been !' but did not. Actual errors:
Traceback (most recent call last):
File "/home/douglasparker/Source/bazel-test-eval/BUILD", line 3
_my_rule(name = 'this_should_fail')
File "/home/douglasparker/Source/bazel-test-eval/build_defs_test.bzl", line 15, in _my_rule_impl
fails_with_backtick_in_message()
File "/home/douglasparker/Source/bazel-test-eval/build_defs.bzl", line 2, in fails_with_backtick_in_message
fail(<1 more arguments>)
Oh noes! Value should have been !
```

It appears that `foo` and `bar` are being treated as a subshell and evaluated inline. I believe the evaluation is coming from [here](https://github.com/bazelbuild/bazel-skylib/blob/560d7b2359aecb066d81041cb532b82d7354561b/toolchains/unittest/BUILD#L35). Running one of the generated shell scripts reproduces the problem:

```
$ bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh
bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: foo: command not found
bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: bar: command not found
bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: bar: command not found
bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh: line 1: foo: command not found
In test _failure_testing_test_impl from //:build_defs_test.bzl: Expected errors to contain 'Oh noes! Value should have been !' but did not. Actual errors:
Traceback (most recent call last):
File "/home/douglasparker/Source/bazel-test-eval/BUILD", line 3
_my_rule(name = 'this_should_fail')
File "/home/douglasparker/Source/bazel-test-eval/build_defs_test.bzl", line 15, in _my_rule_impl
fails_with_backtick_in_message()
File "/home/douglasparker/Source/bazel-test-eval/build_defs.bzl", line 2, in fails_with_backtick_in_message
fail(<1 more arguments>)
Oh noes! Value should have been !
$ cat bazel-bin/failure_testing_test.sh.runfiles/__main__/failure_testing_test.sh
cat << EOF
In test _failure_testing_test_impl from //:build_defs_test.bzl: Expected errors to contain 'Oh noes! Value `foo` should have been `bar`!' but did not. Actual errors:
Traceback (most recent call last):
File "/home/douglasparker/Source/bazel-test-eval/BUILD", line 3
_my_rule(name = 'this_should_fail')
File "/home/douglasparker/Source/bazel-test-eval/build_defs_test.bzl", line 15, in _my_rule_impl
fails_with_backtick_in_message()
File "/home/douglasparker/Source/bazel-test-eval/build_defs.bzl", line 2, in fails_with_backtick_in_message
fail(<1 more arguments>)
Oh noes! Value `bar` should have been `foo`!

EOF
exit 1
```

Contributor guide

Open the contributing guide

Research direction

Start with toolchains/unittest/BUILD around the linked line 35 and inspect the generated failure_testing_test.sh shown in the report. Run the minimal reproduction or generated script on Linux, then verify that an analysistest assertion containing backticks preserves them literally and reports the expected message without shell errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.