cloudnative-pg / cloudnative-pg/ciclops

Bug: `ignoreFailed` tests shouldn't be counted when calculating systematic failures

Open
#12 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
16
Forks
0
Avg merge
11d 4h
Merged PRs (30d)
1

Description

In `compute_systematic_failures_on_metric()` we are counting `ignoreFailed` tests for the `total runs`, which makes it impossible to have systematic failures `by_k8s / by_postgres / by_platform` given that `failures == runs` is always going to be False.

Note: There's only one case in which we could trigger systematic failures `by_k8s / by_postgres / by_platform` at the moment, and that's in case the Ginkgo suite didn't run at all (for example because the Cluster creation or any step that precedes the start of the e2e failed). In that case you would have a single failed `Open Ginkgo report` artifact which would make systematic failures `by_k8s / by_postgres / by_platform` properly trigger.

To fix this we should avoid including `ignoreFailed` tests in the `total runs` when calculating systematic failures.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.