bazelbuild / bazelbuild/bazel-skylib

FR for unittest.bzl: allow testing failing _test rules.

Open
#419 2 comments 3 reactions 0 assignees View on GitHub
Dominant language
Starlark
Stars
444
Forks
202
PR merge metrics
No merged PRs in 30d

Description

Context: https://stackoverflow.com/questions/74844959/how-to-testing-that-bazel-my-test-rule-fails-when-it-should

When building a `custom_test` rule, I'd like to be able to write unit-tests for each of the following cases:

- Passing, builds and produces a success results (possible by just using the rule).
- Fails to build, the rules is used incorrectly (this is what `analysistest.make(..., expect_failure = True)` does).
- Test fails, the rules is used correctly and correctly detects a failure in the thing the rule is designed to test.

The last case is, as best I can tell, not supported but is arguably the most important one to test; a test that passes when it should fail is worse than no test at all.

----

I can kind of fake it by something like this:

```
bad_test(
name = "bad_test_fail_test",
tags = ["manual"],
)

sh_test(
name = "bad_test_failure_test",
srcs = [":expect_fail.sh"], # calls arg and returns error iff arg doesn't
args = ["$(location :bad_test_fail_test)"],
data = [":bad_test_fail_test"],
)
```

But that's a bit of boilerplate and doesn't provide a good way to check that the test failed for the expected reason.

Ideally, I'd like something that can be used like this:

```
load("@bazel_skylib//lib:unittest.bzl", "test_failure_test")

bad_test(
name = "bad_test_fail_test",
tags = ["manual"],
)

test_failure_test(
name = "bad_test_failure_test",
test = [":bad_test_fail_test"],
messages = [
"Your code is bad.",
"Don't feel bad.",
],
)
```

I'm not sure if literals, RE's or both options for the expected patterns would be best.

Contributor guide

Open the contributing guide

Research direction

Start by reading unittest.bzl and the existing analysistest.make(..., expect_failure = True) behavior. Define how a test_failure_test target should execute a failing _test rule and match expected messages, then add coverage for the proposed passing and failing cases. Done means callers can verify that a correctly used test rule fails for the expected reason without the manual sh_test boilerplate.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.