cloudnative-pg / cloudnative-pg/ciclops
Bug: `ignoreFailed` tests shouldn't be counted when calculating systematic failures
- Dominant language
- Python
- Stars
- 16
- Forks
- 0
- Avg merge
- 11d 4h
- Merged PRs (30d)
- 1
Description
In `compute_systematic_failures_on_metric()` we are counting `ignoreFailed` tests for the `total runs`, which makes it impossible to have systematic failures `by_k8s / by_postgres / by_platform` given that `failures == runs` is always going to be False.
Note: There's only one case in which we could trigger systematic failures `by_k8s / by_postgres / by_platform` at the moment, and that's in case the Ginkgo suite didn't run at all (for example because the Cluster creation or any step that precedes the start of the e2e failed). In that case you would have a single failed `Open Ginkgo report` artifact which would make systematic failures `by_k8s / by_postgres / by_platform` properly trigger.
To fix this we should avoid including `ignoreFailed` tests in the `total runs` when calculating systematic failures.
Contributor guide
Assessment
This issue has not been assessed yet.