callstack / callstack/reassure

[BUG] Report results not stable

Open
#513 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
1.5k
Forks
38
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
I have integrated my Reassure tests into a Bitbucket pipeline. My tests run on the pipeline, and I display a summary of my test report in the reports section on Bitbucket. However, the issue is that the tests do not consistently yield the same results when executed.

Reassurance base branch also works. Then it works in the branch where development is carried out and produces the results. This process is done every time a pipeline is triggered.

When I produce the result several times, there is sometimes a big difference between them.

**To Reproduce**
Steps to reproduce the behavior:

Go to the Bitbucket pipeline with Reassure tests.
Run the pipeline.
Again Run the pipeline
Compare the difference between two test results.
See the inconsistency in the test results.
Expected behavior
I expected the tests to produce consistent results every time they are run on the pipeline.

**Screenshots**
![Screenshot 2024-07-11 at 11 09 31](https://github.com/callstack/reassure/assets/19963665/1f674d65-a16e-491c-a86b-2d75eeb3497f)
![Screenshot 2024-07-11 at 11 09 17](https://github.com/callstack/reassure/assets/19963665/8cf2b4f3-d980-46c7-bac8-834ca157d8e1)

As shown in the images, the inconsistency in test results undermines my confidence in Reassure.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.