elastic / elastic/ai-github-actions

[product-manager-impersonator] Failure Signature Buckets for Agent Efficiency Reports

Open
#1,127 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11
Forks
16
Avg merge
22h 9m
Merged PRs (30d)
31

Description

## Feature Idea

**Summary:** Add failure-signature bucketing to Agent Efficiency so weekly failed-run reports automatically group repeated errors instead of requiring manual pattern detection.

## Why a Customer Would Want This
Maintainers running scheduled detector workflows need to triage repeated failures quickly. Today, Agent Efficiency gives raw extracted snippets and asks the agent to infer patterns manually, which is slower and less consistent when the same bootstrap/runtime error repeats across many runs.

## Rough Implementation Sketch
- Extend `scripts/extract-log-errors.py` to emit a grouped `signatures` section (normalized message key, count, affected run IDs/workflows, and one representative snippet) alongside `matches`.
- Update `.github/workflows/agent-efficiency.md` setup/analysis instructions to consume the new grouped section from `/tmp/gh-aw/errors.json` before narrative analysis.
- Add a required “Top failure signatures” subsection in the generated issue format for Agent Efficiency.
- Add tests in `tests/test_extract_log_errors.py` for grouping behavior and stable signature output.

## Why It Won't Be That Hard
This is an additive enhancement on an existing pipeline: the script already centralizes extraction into one JSON payload, and Agent Efficiency already consumes that payload as its primary analysis input. The change is a small extension to one script + one workflow prompt/report template + targeted tests.

## Evidence
- `scripts/extract-log-errors.py` currently emits only a flat list (`total_files_scanned`, `total_matches`, `matches`) with no grouping support (`scripts/extract-log-errors.py:203-207`).
- Agent Efficiency already relies on that output as the pre-analysis source (`.github/workflows/agent-efficiency.md:106-112`, `.github/workflows/agent-efficiency.md:119-130`).
- Agent Efficiency explicitly asks for cross-run pattern identification, which is currently manual (`.github/workflows/agent-efficiency.md:165-174`).
- Recent weekly reports show repeated identical failure signatures across many runs (for example, pre-agent `UV_PATH` bootstrap failure in `#1101` and `#1122`), indicating high value for automatic bucketing.
- This is distinct from existing PM ideas such as `#709` (scheduled workflow health watchdog) and `#1048` (manual detective replay), which target workflow orchestration/replay rather than log-signature aggregation inside Agent Efficiency.

> [!NOTE]
>
> 🔒 Integrity filter blocked 80 items
>
> The following items were blocked because they don't meet the GitHub integrity level.
>
> - [#359](https://github.com/elastic/ai-github-actions/issues/359) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#875](https://github.com/elastic/ai-github-actions/issues/875) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#694](https://github.com/elastic/ai-github-actions/issues/694) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#354](https://github.com/elastic/ai-github-actions/issues/354) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#676](https://github.com/elastic/ai-github-actions/issues/676) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#245](https://github.com/elastic/ai-github-actions/issues/245) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#248](https://github.com/elastic/ai-github-actions/issues/248) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - elastic/ai-github-actions#1067 `list_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - elastic/ai-github-actions#1025 `list_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - elastic/ai-github-actions#875 `list_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#321](https://github.com/elastic/ai-github-actions/issues/321) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#139](https://github.com/elastic/ai-github-actions/issues/139) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#704](https://github.com/elastic/ai-github-actions/issues/704) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#370](https://github.com/elastic/ai-github-actions/issues/370) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#659](https://github.com/elastic/ai-github-actions/pull/659) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#703](https://github.com/elastic/ai-github-actions/pull/703) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - ... and 64 more items
>
> To allow these resources, lower `min-integrity` in your GitHub frontmatter:
>
> ```yaml
> tools:
> github:
> min-integrity: approved # merged | approved | unapproved | none
> ```
>
>

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Trigger Product Manager Impersonator](https://github.com/elastic/ai-github-actions/actions/runs/26030171558)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.