[agent-deep-dive] Agent deep dive: gh-aw-bug-hunter - 2026-09-09

## Agent Deep Dive: `gh-aw-bug-hunter`
**Date:** 2026-09-09
**Runs analyzed:** 8 total (0 success, 8 failure, 0 other)

## Run Outcomes
| Run | Date | Conclusion |
|-----|------|------------|
| [29840755064](https://github.com/elastic/ai-github-actions/actions/runs/29840755064) | 2026-07-21 | failure |
| [29836306664](https://github.com/elastic/ai-github-actions/actions/runs/29836306664) | 2026-07-21 | failure |
| [29836100709](https://github.com/elastic/ai-github-actions/actions/runs/29836100709) | 2026-07-21 | failure |
| [23928108179](https://github.com/elastic/ai-github-actions/actions/runs/23928108179) | 2026-04-03 | failure |
| [23888699772](https://github.com/elastic/ai-github-actions/actions/runs/23888699772) | 2026-04-02 | failure |
| [23862515484](https://github.com/elastic/ai-github-actions/actions/runs/23862515484) | 2026-04-01 | failure |
| [23094261909](https://github.com/elastic/ai-github-actions/actions/runs/23094261909) | 2026-03-14 | failure |
| [23093951187](https://github.com/elastic/ai-github-actions/actions/runs/23093951187) | 2026-03-14 | failure |

## Failure Analysis
All analyzed runs failed with the same observability pattern:

- **Failed point:** log retrieval for each failed run (`/tmp/gh-aw/deep-dive/logs//logs.zip`)
- **Error excerpt:** `{"message":"Not Found","documentation_url":"https://docs.github.com/rest/actions/workflow-runs#download-workflow-run-logs","status":"404"}`
- **Cross-check:** `get_workflow_run_logs_url` also returns `404 Not Found` for all 8 run IDs
- **Additional signal:** `list_workflow_jobs` returns `total_count: 0` for all 8 runs, so no job/step-level traces were retrievable

**Likely cause:** analyzed runs are old enough that logs are no longer available (or were never retrievable), so this deep dive dataset cannot support tool-level behavioral analysis.

## Tool Call Patterns
- **Average turns per run:** unavailable from this dataset (no retrievable agent logs)
- **Most-called tools:** unavailable from this dataset (no `3_agent`/step logs present)
- **Anomalies:**
- 8/8 runs have missing logs (404)
- 8/8 runs have zero retrievable jobs via `list_workflow_jobs`

## Behavioral Observations
What is clear from available data:
- The target workflow (`.github/workflows/gh-aw-bug-hunter.lock.yml`) has a 100% failure rate across all runs returned by the workflow-runs API (8/8).
- The current deep-dive input bundle (`runs.json`, `errors.json`, `logs/`) does not include enough data to inspect agent turn flow, tool usage, or prompt adherence for this workflow.

Secondary evidence from currently open repository issues also points to recent Bug Hunter instability:
- [#2039](https://github.com/elastic/ai-github-actions/issues/2039)
- [#2006](https://github.com/elastic/ai-github-actions/issues/2006)

Both report Bug Hunter runs with missing safe outputs.

## Recurring Patterns
Patterns seen in 2+ runs:
1. `logs.zip` content is a GitHub 404 JSON payload instead of actual workflow logs.
2. `get_workflow_run_logs_url` returns 404 for the same run IDs.
3. `list_workflow_jobs` returns zero jobs for the same run IDs.
4. No successful run exists in the analyzed set to establish a healthy baseline.

## Prompt Improvement Suggestions
Based on the observed failure/observability pattern, here are actionable prompt-level improvements for `gh-aw-bug-hunter`:

1. **Add a strict terminal safe-output checklist at the very end of the prompt.**
- **Observed problem:** recent open failures show runs ending without safe outputs (#2039, #2006).
- **Prompt change:** add an explicit final instruction: “Before finishing, if no issue is being filed, call `noop` with a one-line reason. Never exit without `create_issue` or `noop`.”
- **Expected improvement:** fewer missing-safe-output failures and clearer run outcomes.

2. **Add bounded-turn fallback behavior for dead-end investigations.**
- **Observed problem:** repeated historical failures with no actionable output in analyzed runs.
- **Prompt change:** require a fallback path: after a bounded number of unsuccessful reproduction attempts, emit `noop` with concrete evidence collected so far.
- **Expected improvement:** reduced failure loops, faster completion, and reliable safe-output emission.

3. **Require compact run-summary evidence in final output.**
- **Observed problem:** post-hoc deep dives are blocked when logs are unavailable.
- **Prompt change:** require each run’s final output to include a compact structured summary of key actions (candidate commits checked, reproduction command run, result).
- **Expected improvement:** preserves enough evidence for analysis even when raw logs expire.

4. **Soften unconditional multi-agent fan-out when signal is weak.**
- **Observed problem:** workflow prompt currently mandates “Pick Three, Keep One” for every run, which can increase execution complexity.
- **Prompt change:** make 3-agent fan-out conditional (use only when initial local scan surfaces 2+ credible bug candidates).
- **Expected improvement:** fewer unnecessary tool calls and better reliability on low-signal runs.

> Generated by [Internal: Agent Deep Dive](https://github.com/elastic/ai-github-actions/actions/runs/34366455197) · gpt53codex · 51.8 AIC · ⌖ 3.35 AIC · ⊞ 23.1K · [◷](https://github.com/search?q=repo%3Aelastic%2Fai-github-actions+is%3Aissue+%22gh-aw-workflow-call-id%3A+elastic%2Fai-github-actions%2Fagent-deep-dive%22&type=issues)
> - [x] expires on Sep 23, 2026, 3:03 PM UTC

Start with .github/workflows/gh-aw-bug-hunter.lock.yml and review the prompt behavior around safe outputs, bounded fallback handling, run summaries, and agent fan-out. Compare the workflow with the reported failures in runs 29840755064 and 2039/2006, noting that logs and job traces are unavailable. Done means the workflow reliably emits create_issue or noop and preserves enough final evidence for later analysis.

Written by the indexing model from the issue text.