elastic / elastic/ai-github-actions

Add retry configuration for safe-outputs threat-detection to avoid silent security-check failures

Open
#1,978 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
11
Forks
16
Avg merge
22h 9m
Merged PRs (30d)
31

Description

## Summary

Our `PR Review` agentic workflow (`.github/workflows/gh-aw-pr-review.md` → compiled `gh-aw-pr-review.lock.yml`) uses `safe-outputs` with the default `threat-detection` configuration. We hit a run where the detection step failed with `reason=invalid_report_exhausted exit=2`, even though the underlying Copilot engine had actually invoked the `threat_detection_result` tool with a valid verdict (see [github/gh-aw-threat-detection#1014](https://github.com/github/gh-aw-threat-detection/issues/1014) for the upstream root-cause analysis).

Reference run: https://github.com/elastic/docs-builder/actions/runs/33497222727 (job "run / detection")

## Problem

By design, `gh-aw`'s `threat-detection` defaults `max-retries` to **0** for the detection engine (unlike the main agent, which defaults to 3 retries), specifically because detection is meant to be a bounded scan. However, this means any transient hiccup in the detection engine (harness/tool-call race conditions, momentary API issues, etc.) causes the **entire security check to fail outright with zero retries**, and — because our workflow has `continue-on-error: true` on the detection step (required so a detection failure doesn't block the whole workflow) — the failure is silently swallowed. The overall workflow run reports "success" even though the threat-detection safety net never actually produced a verdict.

This means we currently have **no resilience** against a class of transient detection failures, and no visibility when they occur (the workflow just looks green).

## Request

Please add explicit retry configuration to our `safe-outputs.threat-detection` frontmatter across our `gh-aw`-based workflows (starting with `gh-aw-pr-review`), e.g.:

```yaml
safe-outputs:
threat-detection:
engine:
harness:
max-retries: 2
```

Additionally, please consider:
1. Auditing all reusable workflows in this repo that configure `threat-detection` and adding a sensible non-zero retry count to each, rather than relying on the `gh-aw` default of 0.
2. Adding alerting/visibility (e.g., a step summary annotation or a follow-up issue) when `continue-on-error: true` masks a detection failure, so we don't silently lose the security check without noticing.
3. Documenting this trade-off (continue-on-error + zero retries = silent security-check gaps) in our internal `gh-aw` workflow authoring guidelines.

## References

- gh-aw threat-detection config docs: https://github.github.com/gh-aw/reference/threat-detection/
- Upstream bug report (root cause of the specific failure): github/gh-aw-threat-detection#1014

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.