Harmonize methods used for floating-point approximate equality testing across CCCL test suite
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 486
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 295
Description
Presently, CCCL uses different tests for approximate equality of compute result against a reference value.
CCCL uses the following to test proximity to reference result
$$
\vert a - b \vert \le a_\{\mathrm{tol}} + r_{\mathrm{tol}} \cdot \mathrm{scale}(a, b)
$$
where $a_{\mathrm{tol}}$ and $r_{\mathrm{tol}}$ values used across different constituents of CCCL may differ. Moreover, $\mathrm{scale}(a, b)$ is sometimes defined as $\vert a \vert + \vert b \vert$, and sometimes as $\vert b \vert$.
The latter choice makes the testing function $\mathrm{isclose}(a, b)$ non-symmetric and should be avoided when testing value computed on GPU against value computed on CPU (as opposed to checking for reference value known a-priori).
[PEP-0485](https://peps.python.org/pep-0485/) proposes to use the following test, which is explicitly symmetric and has intuitive meaning of tolerance parameters:
$$
\mathrm{isclose}(a, b) := (\vert a - b \vert \le \max( a_{\mathrm{tol}}, r_{\mathrm{tol}} \cdot \max( \vert a \vert, \vert b \vert )) )
$$
Furthermore, choice of tolerance parameters should depend on `std::numeric_limits::epsilon()` and be based on Wilkinson-type error estimates derived for the pertinent algorithm.
Contributor guide
Assessment
This issue has not been assessed yet.