elastic / elastic/ai-github-actions
[kibana-flaky-test-triage] On-demand workflow to triage failed-test issues by team
- Dominant language
- Python
- Stars
- 11
- Forks
- 16
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 31
Description
## Feature Idea
**Summary:** On-demand workflow that triages open `failed-test` issues in `elastic/kibana` for a specific team. Manually triggered (e.g. monthly), it queries all open flaky test issues for the given team, analyzes each one, and produces a single summary issue with categorized results and fix recommendations.
- Required input: team label (e.g. `Team:Fleet`). Never runs across all teams.
- For each issue: reads the structured issue body (test path, stack trace, buildkite link, codeowners), the test source file, and recent commits touching that file
- AI categorizes root cause and suggests specific fixes (e.g. "add retry to flaky assertion", "mock this HTTP endpoint")
- Output: single summary issue with categorized table of all open flaky tests, per-issue recommendations, and priority signal based on failure frequency
- Pilot with `Team:Fleet`, designed to work for any team label
## Problem
- ~703 `failed-test` issues created or updated per week across Kibana
- Human triage is repetitive: read stack trace, check git blame, decide if flaky vs real
- Most failures fall into well-known patterns (timeouts, network dependencies, race conditions, real regressions)
- EMs have no aggregated view of their team's flaky test debt
- This workflow provides a periodic "state of flaky tests" report with actionable next steps per issue
## Proposed Workflow
1. **Trigger:** Manual workflow dispatch with required `team_label` input (e.g. `Team:Fleet`)
2. **Query:** `gh search issues --repo elastic/kibana --label failed-test --label $team_label --state open`
3. **Parse:** For each issue, extract structured data from the issue body (test ID, file location, stack trace, codeowners, buildkite link)
4. **Enrich:** Read the test source file from `elastic/kibana` and check `git log` for recent commits touching the test file
5. **Analyze:** AI categorizes each issue (open-ended - not a fixed taxonomy) and recommends a specific fix
6. **Report:** Generate a single summary issue with:
- Categorized table of all open flaky tests
- Per-issue root cause analysis and recommended fix
- Failure frequency to help prioritize
## Implementation Considerations
- `failed-test` issue bodies are already well-structured with a metadata table, stack trace, and buildkite link (see evidence below)
- Test file paths are included in the issue body, making enrichment straightforward
- Read-only workflow - analysis and issue creation only, no code changes
- Scoped to one team at a time, keeping context manageable
## Evidence
Example `failed-test` issues from `Team:Fleet` showing the structured format:
- https://github.com/elastic/kibana/issues/258970 - TimeoutError on `page.waitForSelector` (typical timeout flake)
- https://github.com/elastic/kibana/issues/257844 - `"before all"` hook failure in package policy API test
- https://github.com/elastic/kibana/issues/257614 - EPM bulk install prerelease test failure
Each issue contains a structured table (Test ID, Target, Location, Duration, Module, Config path, Code Owners), a stack trace, and a Buildkite build link.
Contributor guide
Assessment
This issue has not been assessed yet.