KhronosGroup / KhronosGroup/SYCL-CTS

Revisit tests with zero assertions

Open
#864 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
76
Forks
97
Avg merge
5d 18h
Merged PRs (30d)
4

Description

There are several test categories that can pass a run while reporting to have zero assertions. This is mostly due to one of two issues: Either the test category still uses the legacy framework for writing tests (which didn't have a concept of assertions, only fail/pass), or because the tests are written in the form of

```c++
if(!evaluateLotsOfThings()) FAIL("Test XY failed")
```

The [atomic](https://github.com/KhronosGroup/SYCL-CTS/blob/3197f433c6cc4a834f73a381df55270346798e05/tests/atomic/atomic_api_32.cpp#L175) tests are an example of the former case, the [address_space](https://github.com/KhronosGroup/SYCL-CTS/blob/3197f433c6cc4a834f73a381df55270346798e05/tests/address_space/address_space_common.h#L213) an example of the latter.

We should strive to have explicit assertions whenever possible, as this makes debugging failing tests much easier. A simple first step for the scenario above would be to change the condition to `CHECK(evaluateLotsOfThings())`. While this doesn't really help debugging, it at least indicates to the reader of a test report that *something* was checked for (otherwise the test may as well have `return`ed on it's first line).

Without having checked each one in detail, this is a list of test categories that I suspect are in need of revisiting:

- address_space
- atomic
- device_event
- error
- exception_handling
- header
- host_task
- kernel_args
- kernel_bundle (has_kernel_bundle_\*, get_kernel_ids_\* and get_kernel_bundle_\* tests)
- pointers
- std_classes
- vector_load_store

Contributor guide

Open the contributing guide

Research direction

Start by inspecting tests/atomic/atomic_api_32.cpp and tests/address_space/address_space_common.h, then review the listed test categories for cases that can pass with zero assertions. The work is done when applicable checks use explicit assertions and the affected tests no longer report zero assertions.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
testing-qa
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.