KhronosGroup / KhronosGroup/SYCL-CTS
Revisit tests with zero assertions
- Dominant language
- C++
- Stars
- 76
- Forks
- 97
- Avg merge
- 5d 18h
- Merged PRs (30d)
- 4
Description
There are several test categories that can pass a run while reporting to have zero assertions. This is mostly due to one of two issues: Either the test category still uses the legacy framework for writing tests (which didn't have a concept of assertions, only fail/pass), or because the tests are written in the form of
```c++
if(!evaluateLotsOfThings()) FAIL("Test XY failed")
```
The [atomic](https://github.com/KhronosGroup/SYCL-CTS/blob/3197f433c6cc4a834f73a381df55270346798e05/tests/atomic/atomic_api_32.cpp#L175) tests are an example of the former case, the [address_space](https://github.com/KhronosGroup/SYCL-CTS/blob/3197f433c6cc4a834f73a381df55270346798e05/tests/address_space/address_space_common.h#L213) an example of the latter.
We should strive to have explicit assertions whenever possible, as this makes debugging failing tests much easier. A simple first step for the scenario above would be to change the condition to `CHECK(evaluateLotsOfThings())`. While this doesn't really help debugging, it at least indicates to the reader of a test report that *something* was checked for (otherwise the test may as well have `return`ed on it's first line).
Without having checked each one in detail, this is a list of test categories that I suspect are in need of revisiting:
- address_space
- atomic
- device_event
- error
- exception_handling
- header
- host_task
- kernel_args
- kernel_bundle (has_kernel_bundle_\*, get_kernel_ids_\* and get_kernel_bundle_\* tests)
- pointers
- std_classes
- vector_load_store
Contributor guide
Research direction
Start by inspecting tests/atomic/atomic_api_32.cpp and tests/address_space/address_space_common.h, then review the listed test categories for cases that can pass with zero assertions. The work is done when applicable checks use explicit assertions and the affected tests no longer report zero assertions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100