github / github/gh-aw-threat-detection
threat-detect reports invalid_report_exhausted despite successful threat_detection_result tool call
- Dominant language
- Go
- Stars
- 13
- Forks
- 7
- Avg merge
- 9h 52m
- Merged PRs (30d)
- 25
Description
## Summary
`threat-detect` (v0.5.1, `--engine copilot`) fails with `reason=invalid_report_exhausted exit=2` even though the Copilot engine successfully invoked the `threat_detection_result` tool with a valid verdict. This causes the detection step to fail and, when `continue-on-error: true` is set on the calling workflow, silently disables the safety check while the overall job still reports "success".
## Observed in
Workflow run: https://github.com/elastic/docs-builder/actions/runs/33497222727 (job "run / detection")
Repro engine: `copilot` / model `gpt-5.3-codex`
## Relevant log excerpt
```
[threat-detect] run start: version=v0.5.1 engine=copilot model=gpt-5.3-codex retries=0 max_turns=50 engine_timeout=5m0s
...
[threat-detect] detection attempt 1 of 1
[engine] `threat_detection_result --prompt-injection false --secret-leak false --malicious-patch false`
...
[engine] [copilot-harness] success on attempt 1: totalDuration=3s
[threat-detect] attempt 1 outcome=no_verdict err=open /tmp/threat-detect-result-545100967.json: no such file or directory
Error running detection: detection model did not record a usable verdict via the threat_detection_result tool after 1 attempt(s): open /tmp/threat-detect-result-545100967.json: no such file or directory
THREAT_DETECTION_STATUS: reason=invalid_report_exhausted exit=2
```
This same failure signature repeated on a second attempt in the same run.
## Root cause (suspected)
The Copilot engine harness correctly invoked the `threat_detection_result` tool and reported success (`exitCode=0`), but `threat-detect` looked for the tool-call result at a temp path (`/tmp/threat-detect-result-.json`) that was never written / not found. This looks like a path mismatch or race condition between when the Copilot harness records the tool call output and when `threat-detect` reads it back, rather than an actual "invalid report" from the model.
## Impact
- The detection job fails (`ERR_PARSE: Detection result file not found`), but because `GH_AW_DETECTION_CONTINUE_ON_ERROR=true` is set by the calling workflow, the overall job/run still reports **success**.
- This means the prompt-injection / secret-leak / malicious-patch safety check silently produced no verdict for this PR, even though the model correctly determined there were no threats — the report path just wasn't propagated.
## Suggested fixes
1. Investigate why the tool-call result file (`/tmp/threat-detect-result-.json`) isn't found after the Copilot harness reports a successful `threat_detection_result` call — likely a race condition or incorrect file path resolution between the harness and `threat-detect`.
2. Consider surfacing a more visible warning/annotation when `no_verdict` occurs despite a successful engine run, distinguishing it from a genuine "model refused to report" case.
3. Consider retrying more than 0 times (`retries=0` in the log) before exhausting, or making retry count configurable per engine.
## Environment
- `gh-aw` version: v0.28.10 (`GH_AW_INFO_AWF_VERSION`)
- Engine: Copilot CLI `1.0.80` (`GH_AW_INFO_VERSION`)
- Runner: ubuntu-24.04, 20260823.283.1
Contributor guide
Research direction
Start with the threat-detect and Copilot harness code paths that handle the successful threat_detection_result call and the temporary /tmp/threat-detect-result-.json path. Reproduce the linked workflow failure and trace how the tool-call output is written and read after the harness reports success. Done means a valid verdict is consumed without invalid_report_exhausted, and no-verdict failures are visibly distinguished from genuine model refusals.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, go
- Domain
- ci-cd, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100