bazelbuild / bazelbuild/bazel-skylib
FR for unittest.bzl: allow testing failing _test rules.
- Dominant language
- Starlark
- Stars
- 444
- Forks
- 202
- PR merge metrics
- No merged PRs in 30d
Description
Context: https://stackoverflow.com/questions/74844959/how-to-testing-that-bazel-my-test-rule-fails-when-it-should
When building a `custom_test` rule, I'd like to be able to write unit-tests for each of the following cases:
- Passing, builds and produces a success results (possible by just using the rule).
- Fails to build, the rules is used incorrectly (this is what `analysistest.make(..., expect_failure = True)` does).
- Test fails, the rules is used correctly and correctly detects a failure in the thing the rule is designed to test.
The last case is, as best I can tell, not supported but is arguably the most important one to test; a test that passes when it should fail is worse than no test at all.
----
I can kind of fake it by something like this:
```
bad_test(
name = "bad_test_fail_test",
tags = ["manual"],
)
sh_test(
name = "bad_test_failure_test",
srcs = [":expect_fail.sh"], # calls arg and returns error iff arg doesn't
args = ["$(location :bad_test_fail_test)"],
data = [":bad_test_fail_test"],
)
```
But that's a bit of boilerplate and doesn't provide a good way to check that the test failed for the expected reason.
Ideally, I'd like something that can be used like this:
```
load("@bazel_skylib//lib:unittest.bzl", "test_failure_test")
bad_test(
name = "bad_test_fail_test",
tags = ["manual"],
)
test_failure_test(
name = "bad_test_failure_test",
test = [":bad_test_fail_test"],
messages = [
"Your code is bad.",
"Don't feel bad.",
],
)
```
I'm not sure if literals, RE's or both options for the expected patterns would be best.
Contributor guide
Research direction
Start by reading unittest.bzl and the existing analysistest.make(..., expect_failure = True) behavior. Define how a test_failure_test target should execute a failing _test rule and match expected messages, then add coverage for the proposed passing and failing cases. Done means callers can verify that a correctly used test rule fails for the expected reason without the manual sh_test boilerplate.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100