elm-explorations / elm-explorations/test

Feature request: Add a way to assert that a test fails

Open
#186 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Elm
Stars
244
Forks
40
Avg merge
1d 23h
Merged PRs (30d)
2

Description

## Context

In `elm-review` I provide a [testing module](https://package.elm-lang.org/packages/jfmengels/elm-review/latest/Review-Test) that lets you run a lot of custom assertions so you don't have to write them themselves.

The problem with this testing module is that I can't test it. I can't write a test that says "in this scenario, I expect the testing module to cause a test failure that fails with this error message"

At the moment, I have to do manual testing to make sure that the test fails as expected (meaning I don't have a net to catch me if I ever mess it up). I also write separate tests to make sure that the error messages look as expected, but I'm basically testing the error message function, not the assertion itself (https://github.com/jfmengels/elm-review/blob/master/tests/Review/Test/FailureMessageTest.elm).

## Proposal

I would like to propose an `Expect.toFail` function.

```elm
{-| Expect the test to fail with the given message.

This is useful if you wish to ensure that custom test assertions work as expected.
-}
toFail : (String -> Expectation) -> Expectation -> Expectation
```

### Usage

```elm
test "..." <|
\() ->
input
|> someTestUtility
-- Have this test pass if the test "so far" fails
|> Expect.toFail (\_ -> Expect.pass)

test "..." <|
\() ->
input
|> someTestUtility
-- Have this test pass if the test "so far" fails with a message that contains "incorrect location"
-- and have it fail otherwise.
|> Expect.toFail (\error -> String.contains "incorrect location" test |> Expect.equal True)
```

I think it would be possible to have the function compare the resulting error and match it against the expected error string, but I feel that this is more general and powerful.

In my use-case, I will likely compare the error with some exact string, but I will also make sure that all lines in the error message are less than 76 characters long, otherwise the output is really ugly when presented to the user. I will likely do something like this:
```elm
test "..." <|
\() ->
input
|> someTestUtility
-- Have this test pass if the test "so far" fails with a message that contains "incorrect location"
-- and have it fail otherwise.
|> Expect.toFail (expectMessageEqual "....")

-- Directly taken from
-- https://github.com/jfmengels/elm-review/blob/master/tests/Review/Test/FailureMessageTest.elm#L45-L59
expectMessageEqual : String -> String -> Expectation
expectMessageEqual expectedMessage =
Expect.all
[ Expect.equal <| String.trim expectedMessage
, \receivedMessage ->
Expect.all
(String.lines receivedMessage
|> List.map
(\line () ->
(String.length line <= 76)
|> Expect.true ("Message has line longer than 76 characters:\n\n" ++ line)
)
)
()
]
```

Alternatively, the first argument could be `(String -> Bool)` but it could be nice to have the same API, and also to have nice error messages. If we do go with `(String -> Expectation)`, there is a need to figure out what the error message would look like when wrapped inside a `Expect.toFail`.

## Afterword

Having this available would help me be a lot more confident about this testing module that I provide to my users. Currently, this is quite a lot of work to separate the error message from the test failures, and it is still somewhat unreliable and requires manual testing.

I don't know if it would make sense to have a corresponding `Expect.toSucceed` but I don't believe I have a need for it.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.