[agent-deep-dive] Agent deep dive: gh-aw-issue-triage - 2026-09-11

## Agent Deep Dive: `gh-aw-issue-triage`
**Date:** 2026-09-11
**Runs analyzed:** 20 total (0 success, 11 failure, 9 other)

## Run Outcomes
| Run | Date | Conclusion |
|-----|------|------------|
| [29840762333](https://github.com/elastic/ai-github-actions/actions/runs/29840762333) | 2026-07-21 | failure |
| [29836329748](https://github.com/elastic/ai-github-actions/actions/runs/29836329748) | 2026-07-21 | failure |
| [29836139907](https://github.com/elastic/ai-github-actions/actions/runs/29836139907) | 2026-07-21 | failure |
| [28094153971](https://github.com/elastic/ai-github-actions/actions/runs/28094153971) | 2026-06-24 | action_required |
| [28093170555](https://github.com/elastic/ai-github-actions/actions/runs/28093170555) | 2026-06-24 | failure |
| [28091090865](https://github.com/elastic/ai-github-actions/actions/runs/28091090865) | 2026-06-24 | failure |
| [28023210330](https://github.com/elastic/ai-github-actions/actions/runs/28023210330) | 2026-06-23 | action_required |
| [28022588245](https://github.com/elastic/ai-github-actions/actions/runs/28022588245) | 2026-06-23 | failure |
| [27942881476](https://github.com/elastic/ai-github-actions/actions/runs/27942881476) | 2026-06-22 | failure |
| [27817041997](https://github.com/elastic/ai-github-actions/actions/runs/27817041997) | 2026-06-19 | action_required |
| [27816451993](https://github.com/elastic/ai-github-actions/actions/runs/27816451993) | 2026-06-19 | action_required |
| [27678085624](https://github.com/elastic/ai-github-actions/actions/runs/27678085624) | 2026-06-17 | action_required |
| [27677958830](https://github.com/elastic/ai-github-actions/actions/runs/27677958830) | 2026-06-17 | failure |
| [27626018806](https://github.com/elastic/ai-github-actions/actions/runs/27626018806) | 2026-06-16 | action_required |
| [27625671466](https://github.com/elastic/ai-github-actions/actions/runs/27625671466) | 2026-06-16 | failure |
| [27625333530](https://github.com/elastic/ai-github-actions/actions/runs/27625333530) | 2026-06-16 | cancelled |
| [27623843021](https://github.com/elastic/ai-github-actions/actions/runs/27623843021) | 2026-06-16 | failure |
| [27616807301](https://github.com/elastic/ai-github-actions/actions/runs/27616807301) | 2026-06-16 | action_required |
| [27616656184](https://github.com/elastic/ai-github-actions/actions/runs/27616656184) | 2026-06-16 | failure |
| [27603886981](https://github.com/elastic/ai-github-actions/actions/runs/27603886981) | 2026-06-16 | action_required |

## Failure Analysis
Observed failed-run logs (7 runs with log artifacts) show one dominant failure mode in the `agent` job:

- **Failed step:** agent execution (`3_agent.txt`)
- **Error excerpt (recurs):** `BYOK providers require an explicit model. Run \`copilot help providers\` for configuration details.`
- **Evidence:**
- `/tmp/gh-aw/deep-dive/logs/28093170555/3_agent.txt:1977`
- `/tmp/gh-aw/deep-dive/logs/28091090865/3_agent.txt:1870`
- `/tmp/gh-aw/deep-dive/logs/28022588245/3_agent.txt:2084`
- `/tmp/gh-aw/deep-dive/logs/27942881476/3_agent.txt:2088`
- `/tmp/gh-aw/deep-dive/logs/27677958830/3_agent.txt:2104`
- `/tmp/gh-aw/deep-dive/logs/27625671466/3_agent.txt:2092`
- `/tmp/gh-aw/deep-dive/logs/27623843021/3_agent.txt:1990`
- **Termination pattern:** each run exhausts retries and exits in the same step (`all 3 retries exhausted`, then `Process completed with exit code 1`).
- **Likely cause:** model is not being passed through to the Copilot CLI launch in this execution path (spawn line shows no explicit model argument), despite workflow defaults defining one (`.github/workflows/gh-aw-issue-triage.md:20,27-31`).

Additional gap:
- 4 failed runs in the analyzed window have no downloadable deep-dive log bundle here (`29840762333`, `29836329748`, `29836139907`, `27616656184`), which limits failure-forensics completeness.

## Tool Call Patterns
- Average turns per run: **0 observed agent turns** in all 7 failed runs with logs (failure occurs before usable conversation/tool loop).
- Most-called tools: **none in agent execution phase** (startup fails before task-level tool usage).
- Anomalies:
- Deterministic startup error retried 3 times per run (4 total launch attempts), adding latency without changing outcome.
- Conclusion step marks `Missing tool report-as-failure: true` and `Missing data report-as-failure: true` with `Agent output content length: 13` in repeated failures (example: `/tmp/gh-aw/deep-dive/logs/28093170555/0_conclusion.txt:458-460`).

## Behavioral Observations
- What it does well:
- Runtime setup, MCP gateway startup, and connectivity checks are consistently successful before agent launch.
- Failure-reporting pipeline is active and appends to existing tracking issue (example: `/tmp/gh-aw/deep-dive/logs/28093170555/0_conclusion.txt:465-470`).
- Where it struggles:
- The core triage task never starts when BYOK requires explicit model selection.
- Retry policy treats this as transient (`partial_execution`) even though the error text is deterministic and configuration-related.

## Recurring Patterns
Patterns seen in 2+ runs:
1. Identical BYOK explicit-model error in every failed run with logs (7/7).
2. Identical retry sequence (4 launch attempts, then fail) in every failed run with logs (7/7).
3. Identical conclusion flags around missing tool/data reporting and tiny output length in every failed run with logs (7/7).

## Prompt Improvement Suggestions
1. **Add an explicit model preflight gate before Copilot launch**
- Observed problem: repeated `BYOK providers require an explicit model` startup failure.
- Prompt/workflow change: add a hard preflight validation that fails fast when model resolution is empty/invalid, and logs the resolved model value/source.
- Expected improvement: prevents wasted retries, makes root cause obvious in first failure line.

2. **Treat BYOK explicit-model error as non-retriable**
- Observed problem: deterministic error retried 3 times per run.
- Prompt/runtime change: classify this exact error string as configuration-fatal and stop after first attempt.
- Expected improvement: ~3x fewer failed launch attempts, faster feedback, lower run cost.

3. **Surface the resolved model in run summary and failure issue body**
- Observed problem: failure report includes many flags but not the resolved/attempted model.
- Prompt/workflow change: include resolved model, provider, and launch args (sanitized) in conclusion output.
- Expected improvement: easier triage, faster fix verification, less manual log digging.

4. **Ensure deep-dive artifacts include logs for all failed runs in scope**
- Observed problem: 4 failed runs lacked log bundles in this deep-dive payload.
- Prompt/pipeline change: require artifact completeness check before analysis task starts; if missing, report explicit missing run IDs upfront.
- Expected improvement: complete forensic coverage and higher-confidence reporting.

> Generated by [Internal: Agent Deep Dive](https://github.com/elastic/ai-github-actions/actions/runs/34612747516) · gpt53codex · 66.1 AIC · ⌖ 4.41 AIC · ⊞ 23.1K · [◷](https://github.com/search?q=repo%3Aelastic%2Fai-github-actions+is%3Aissue+%22gh-aw-workflow-call-id%3A+elastic%2Fai-github-actions%2Fagent-deep-dive%22&type=issues)
> - [x] expires on Sep 25, 2026, 3:00 PM UTC

Start with .github/workflows/gh-aw-issue-triage.md at lines 20 and 27-31, then compare its model configuration with the Copilot launch shown in the referenced 3_agent.txt logs. Verify the repeated BYOK failure and missing artifacts across the listed runs. Done means the execution path validates and passes an explicit model, handles this configuration failure appropriately, and reports missing deep-dive data clearly.

Written by the indexing model from the issue text.