catchorg / catchorg/Catch2

How to add custom output on test failure (for binary data comparison)

Open
#1,116 10 comments 1 reaction 0 assignees View on GitHub
Query
Dominant language
C++
Stars
21.5k
Forks
3.5k
Avg merge
3d 16h
Merged PRs (30d)
2

Description

## Description

I would like to port my tests away from Boost.Test and I have started investigating Catch. Everything looks good except I can't work out how to cleanly compare binary data with Catch. Because my code produces binary files, if a test fails because the output data is not what was expected, I need to see the offending data in the test output so I can work out what went wrong.

### Steps to reproduce
```
TEST_CASE("Binary data comparison") {
std::string a( "test\x00\x01\x02\x03ing!", 12 );
std::string b( "test\x03\x02\x01\x00ing!", 12 );

REQUIRE( a == b );
}
```
Output:
```
tests_catch.cpp:69: FAILED:
REQUIRE( a == b )
with expansion:
"testing!" == "testing!"
```
The binary data is not present in this format, and if it were, it would not be as clear as it could be.

### Extra information

Boost.Test lets me write a custom output function, so I wrote one that looks like this, using ANSI codes to colour the differing bytes:
```
test.cpp(566): error: in "test_xyz/test_open[xyz]": Example failure.
Exp: 74 65 73 74 00 01 02 03 69 6e 67 21 test....ing!
Got: 74 65 73 74 03 02 01 00 69 6e 67 21 test....ing!
```
![test-example](https://user-images.githubusercontent.com/171656/33235873-d18c6b50-d28e-11e7-9e22-2d33a8a61ad8.png)

Is it possible to do the same with Catch?

Contributor guide

Open the contributing guide

Research direction

Reproduce the binary comparison in tests_catch.cpp using the shown REQUIRE(a == b) case and compare its output with the requested byte-oriented format. Read the existing assertion-reporting and custom-output documentation or entry points, if available, to determine whether this needs a supported extension or a new feature. Done means the framework can present the differing binary data clearly, including the relevant bytes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.