eic / eic/epic-capybara

Emit non-zero exit code when differences

Open
#1 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
1
Avg merge
2d 2h
Merged PRs (30d)
4

Description

**Is your feature request related to a problem? Please describe.**
For more automation, a non-zero exit code when there are differences can be helpful. This can be used to require no differences to be present, or (if the non-zero exit code contains information) to require less than a certain 'level' of differences.

**Describe the solution you'd like**
Example could be (to be supported by other new functionality):
- 0: no differences,
- >0: differences:
- 1: no differences in number of entries in collections, and differences in values are within numerical precision (specified or assumed),
- 2: differences in number of entries (Poisson) and value cumulative distributions (Kolmogorov-Smirnov) are statistically insignificant (1% CL? to allow use in CI),
- 3: differences in number of entries or value cumulative distributions are statistically significant.

**Describe alternatives you've considered**
This could also be parsed from text output.

**Additional context**
We may not want to fail every job and pipeline that has non-zero, but this could lead to e.g. an info message when 1, a warning when 2, and a fail when 3.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.