apache / apache/incubator-seata

Introduce automated flaky test tracking like OpenSearch

Open
#7,448 3 comments 0 reactions 1 assignee Claimed by @OmCheeLin View on GitHub
task: help-wanted
Dominant language
Java
Stars
26k
Forks
8.8k
Avg merge
1d 8h
Merged PRs (30d)
4

Description

### Why you need it?

I would like to suggest introducing an automated flaky test tracking mechanism, similar to what [OpenSearch](https://github.com/opensearch-project/OpenSearch/issues/14308) has adopted.

### How it could be?

### Proposal
Introduce a similar system to automatically report flaky tests in Seata. This would help:
- Systematically track flaky behavior
- Prioritize test stability improvements
- Avoid merging regressions masked by inconsistent test results

### Implementation Ideas

- Hook into existing CI test result processing
- Identify tests that fail inconsistently across retries
- Auto-generate GitHub issues using a bot or GitHub Actions
- Apply consistent labels (e.g., flaky-test, test-failure, component:)

### Benefits
- Easier triage of test failures
- More transparent test health history
- Helps contributors focus on test reliability

Thanks 🙏

### Other related information

- https://learn.microsoft.com/en-us/azure/devops/report/powerbi/sample-test-analytics-flaky-tests?view=azure-devops&tabs=powerbi

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.