Nexus: Improve verbosity in testing
- Dominant language
- C++
- Stars
- 403
- Forks
- 154
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 82
Description
## Problem
A problem I've experienced and heard others such as @prckent talk about is the lack of verbose output when running Nexus tests. Currently, when a test fails in Nexus, the error that gets raised is an `AssertionError`, which only points to the point in the testing module where the `assert...` expression is, but this doesn't provide any extra information about what actually failed. Often the `assert...` expression looks something like this:
```python
def test_files():
filenames = get_filenames()
files = get_files()
assert(set(files.keys())==set(filenames))
#end def test_files
```
and if the test fails there is no indication of what actually is different between the objects. It is then usually up to the tester to go in and modify the code and do some level of debugging. This can take a lot of time, and if a test is run in an environment that requires a complete build of QMCPACK (e.g. the current GitHub workflows), those tests can take a lot of time to rerun.
## Proposed Solution
I think the ideal solution here is to take a page out of the [Rust playbook](https://doc.rust-lang.org/book/ch11-01-writing-tests.html#testing-equality-with-assert_eq-and-assert_ne) and provide a custom `assert...` function (like the Rust `assert_eq!` macro) that not only performs the assert, but also in the case of failure will print the values that did not equal each other.
### General Form
```python
def assert_eq(val1, val2):
try:
assert(val1 == val2)
except AssertionError:
print("\n\n")
print("Value 1 does not equal Value 2!")
print("="*50)
print("Value 1: ")
print(val1)
print("="*50)
print("Value 2: ")
print(val2)
print("="*50)
print("\n\n")
raise AssertionError
#end def assert_eq
```
The first thing that happens is an attempt to assert equality, nested in a `try... except` block. Upon failure, the test accepts the `AssertionError` (specifically it should only accept `AssertionError` so nothing else accidentally gets caught), then prints out some space and separators to make the values distinct from the rest of the text output. After printing the values, it then re-raises the `AssertionError` so that it behaves in the same way as a normal `assert...` statement.
Obviously the equality in the assertion can change, and there potentially could be specific forms of the assert for specific datatypes (or a single assert that brings all datatypes under its umbrella and selects the correct instance), but this is the general idea for how it would look when it's implemented.
I'm happy to hear feedback and get some good discussion going for this!
Contributor guide
Assessment
This issue has not been assessed yet.