ctrf-io / ctrf-io/github-test-reporter

Flaky Tests not getting reported

Open
#253 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
376
Forks
42
PR merge metrics
No merged PRs in 30d

Description

Hi,
I am not able to get the Flaky Test Report, I uploaded test reports with both Passed and Failed for same test manually.
But it is not reported as Flaky, please help me to resolve this issue.

I have attached the xml and the workflow too

Single Test Failure:

<testcase name="testButtonClickAdd" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="1.438" />
<testcase name="testButtonClickSubtract" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.06" >
    <failure>androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'with text: is "21"'
    </failure>
</testcase>
<testcase name="testButtonClickMultiply" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.38" >
</testcase>
Image




Double Test Failure:

<testcase name="testButtonClickAdd" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="1.438" />
<testcase name="testButtonClickSubtract" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.06" >
    <failure>androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'with text: is "21"' 
    </failure>
</testcase>
<testcase name="testButtonClickMultiply" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.06" >
    <failure>androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'with text: is "21"'
    </failure>
</testcase>
Image



Yaml File (Wokflow)

    - name: Publish Test Report with Insights
      uses: ctrf-io/github-test-reporter@v1
      with:
        report-path: 'sample.json'
        summary-delta-report: true
        insights-report: true
        flaky-report: true
        flaky-rate-report: true
        fail-rate-report: true
        slowest-report: true
        previous-results-report: true
        fetch-previous-results: true
        upload-artifact: true
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      if: always()

Can u please provide the Workflow to report Flaky Test and also please provide the sample xml

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported workflow using ctrf-io/github-test-reporter@v1, the sample.json path, and the attached single- and double-failure XML examples. Check whether the generated flaky report identifies a test present in both results, and consider the issue done when a documented workflow and sample XML produce the expected flaky-test report.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.