ctrf-io / ctrf-io/github-test-reporter
Flaky Tests not getting reported
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 376
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I am not able to get the Flaky Test Report, I uploaded test reports with both Passed and Failed for same test manually.
But it is not reported as Flaky, please help me to resolve this issue.
I have attached the xml and the workflow too
Single Test Failure:
<testcase name="testButtonClickAdd" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="1.438" />
<testcase name="testButtonClickSubtract" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.06" >
<failure>androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'with text: is "21"'
</failure>
</testcase>
<testcase name="testButtonClickMultiply" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.38" >
</testcase>
Double Test Failure:
<testcase name="testButtonClickAdd" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="1.438" />
<testcase name="testButtonClickSubtract" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.06" >
<failure>androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'with text: is "21"'
</failure>
</testcase>
<testcase name="testButtonClickMultiply" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.06" >
<failure>androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'with text: is "21"'
</failure>
</testcase>
Yaml File (Wokflow)
- name: Publish Test Report with Insights
uses: ctrf-io/github-test-reporter@v1
with:
report-path: 'sample.json'
summary-delta-report: true
insights-report: true
flaky-report: true
flaky-rate-report: true
fail-rate-report: true
slowest-report: true
previous-results-report: true
fetch-previous-results: true
upload-artifact: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
if: always()
Can u please provide the Workflow to report Flaky Test and also please provide the sample xml
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported workflow using ctrf-io/github-test-reporter@v1, the sample.json path, and the attached single- and double-failure XML examples. Check whether the generated flaky report identifies a test present in both results, and consider the issue done when a documented workflow and sample XML produce the expected flaky-test report.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, typescript
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100