Emit non-zero exit code when differences
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 4
Description
**Is your feature request related to a problem? Please describe.**
For more automation, a non-zero exit code when there are differences can be helpful. This can be used to require no differences to be present, or (if the non-zero exit code contains information) to require less than a certain 'level' of differences.
**Describe the solution you'd like**
Example could be (to be supported by other new functionality):
- 0: no differences,
- >0: differences:
- 1: no differences in number of entries in collections, and differences in values are within numerical precision (specified or assumed),
- 2: differences in number of entries (Poisson) and value cumulative distributions (Kolmogorov-Smirnov) are statistically insignificant (1% CL? to allow use in CI),
- 3: differences in number of entries or value cumulative distributions are statistically significant.
**Describe alternatives you've considered**
This could also be parsed from text output.
**Additional context**
We may not want to fail every job and pipeline that has non-zero, but this could lead to e.g. an info message when 1, a warning when 2, and a fail when 3.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.