bazelbuild / bazelbuild/bazel-skylib

Prevent printing output to allow failure testing of custom asserts

Open
#346 0 comments 0 reactions 0 assignees View on GitHub
P3 type: feature request
Dominant language
Starlark
Stars
444
Forks
202
PR merge metrics
No merged PRs in 30d

Description

It's currently a bit tricky to do failure tests of custom assert functions. One particular wart is that any expected failure will call print() and output a bogus failure message, even though the test passes. e.g.

```
def assert_even(env, n):
asserts.true(env, n %2 == 0)

def test_custom_assert(ctx):
env = unittest.begin(ctx)
assert_even(env, 1)

env.failures.pop() # prevent failure from propagating
return unittest.end(ctx)
```

The above will pass, as expected, but print an bogus error. The logic for this is in `unittest.bzl#_fail`:

```
def _fail(env, msg):
print(full_msg)
env.failures.append(full_msg)
```

Here's a couple proposed solutions:
1. Get the "print" function from the env struct. This allow callers to mock it out with their own callable.
2. Add a "bool print_on_failure" optional attribute to the env struct. This would skip calling print() if true.

Contributor guide

Open the contributing guide

Research direction

The relevant logic is in unittest.bzl#_fail; start by reading how custom assertions record failures and inspect the existing unittest tests. Done means a custom-assert failure can be verified through the recorded environment without producing the bogus output, with regression coverage for that behavior.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, testing
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.