elm-explorations / elm-explorations/test
Feature request: Add a way to assert that a test fails
- Dominant language
- Elm
- Stars
- 244
- Forks
- 40
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 2
Description
## Context
In `elm-review` I provide a [testing module](https://package.elm-lang.org/packages/jfmengels/elm-review/latest/Review-Test) that lets you run a lot of custom assertions so you don't have to write them themselves.
The problem with this testing module is that I can't test it. I can't write a test that says "in this scenario, I expect the testing module to cause a test failure that fails with this error message"
At the moment, I have to do manual testing to make sure that the test fails as expected (meaning I don't have a net to catch me if I ever mess it up). I also write separate tests to make sure that the error messages look as expected, but I'm basically testing the error message function, not the assertion itself (https://github.com/jfmengels/elm-review/blob/master/tests/Review/Test/FailureMessageTest.elm).
## Proposal
I would like to propose an `Expect.toFail` function.
```elm
{-| Expect the test to fail with the given message.
This is useful if you wish to ensure that custom test assertions work as expected.
-}
toFail : (String -> Expectation) -> Expectation -> Expectation
```
### Usage
```elm
test "..." <|
\() ->
input
|> someTestUtility
-- Have this test pass if the test "so far" fails
|> Expect.toFail (\_ -> Expect.pass)
test "..." <|
\() ->
input
|> someTestUtility
-- Have this test pass if the test "so far" fails with a message that contains "incorrect location"
-- and have it fail otherwise.
|> Expect.toFail (\error -> String.contains "incorrect location" test |> Expect.equal True)
```
I think it would be possible to have the function compare the resulting error and match it against the expected error string, but I feel that this is more general and powerful.
In my use-case, I will likely compare the error with some exact string, but I will also make sure that all lines in the error message are less than 76 characters long, otherwise the output is really ugly when presented to the user. I will likely do something like this:
```elm
test "..." <|
\() ->
input
|> someTestUtility
-- Have this test pass if the test "so far" fails with a message that contains "incorrect location"
-- and have it fail otherwise.
|> Expect.toFail (expectMessageEqual "....")
-- Directly taken from
-- https://github.com/jfmengels/elm-review/blob/master/tests/Review/Test/FailureMessageTest.elm#L45-L59
expectMessageEqual : String -> String -> Expectation
expectMessageEqual expectedMessage =
Expect.all
[ Expect.equal <| String.trim expectedMessage
, \receivedMessage ->
Expect.all
(String.lines receivedMessage
|> List.map
(\line () ->
(String.length line <= 76)
|> Expect.true ("Message has line longer than 76 characters:\n\n" ++ line)
)
)
()
]
```
Alternatively, the first argument could be `(String -> Bool)` but it could be nice to have the same API, and also to have nice error messages. If we do go with `(String -> Expectation)`, there is a need to figure out what the error message would look like when wrapped inside a `Expect.toFail`.
## Afterword
Having this available would help me be a lot more confident about this testing module that I provide to my users. Currently, this is quite a lot of work to separate the error message from the test failures, and it is still somewhat unreliable and requires manual testing.
I don't know if it would make sense to have a corresponding `Expect.toSucceed` but I don't believe I have a need for it.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.