github / github/gh-aw-threat-detection

threat-detect reports invalid_report_exhausted despite successful threat_detection_result tool call

Open
#1,014 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13
Forks
7
Avg merge
9h 52m
Merged PRs (30d)
25

Description

## Summary

`threat-detect` (v0.5.1, `--engine copilot`) fails with `reason=invalid_report_exhausted exit=2` even though the Copilot engine successfully invoked the `threat_detection_result` tool with a valid verdict. This causes the detection step to fail and, when `continue-on-error: true` is set on the calling workflow, silently disables the safety check while the overall job still reports "success".

## Observed in

Workflow run: https://github.com/elastic/docs-builder/actions/runs/33497222727 (job "run / detection")
Repro engine: `copilot` / model `gpt-5.3-codex`

## Relevant log excerpt

```
[threat-detect] run start: version=v0.5.1 engine=copilot model=gpt-5.3-codex retries=0 max_turns=50 engine_timeout=5m0s
...
[threat-detect] detection attempt 1 of 1
[engine] `threat_detection_result --prompt-injection false --secret-leak false --malicious-patch false`
...
[engine] [copilot-harness] success on attempt 1: totalDuration=3s
[threat-detect] attempt 1 outcome=no_verdict err=open /tmp/threat-detect-result-545100967.json: no such file or directory
Error running detection: detection model did not record a usable verdict via the threat_detection_result tool after 1 attempt(s): open /tmp/threat-detect-result-545100967.json: no such file or directory
THREAT_DETECTION_STATUS: reason=invalid_report_exhausted exit=2
```

This same failure signature repeated on a second attempt in the same run.

## Root cause (suspected)

The Copilot engine harness correctly invoked the `threat_detection_result` tool and reported success (`exitCode=0`), but `threat-detect` looked for the tool-call result at a temp path (`/tmp/threat-detect-result-.json`) that was never written / not found. This looks like a path mismatch or race condition between when the Copilot harness records the tool call output and when `threat-detect` reads it back, rather than an actual "invalid report" from the model.

## Impact

- The detection job fails (`ERR_PARSE: Detection result file not found`), but because `GH_AW_DETECTION_CONTINUE_ON_ERROR=true` is set by the calling workflow, the overall job/run still reports **success**.
- This means the prompt-injection / secret-leak / malicious-patch safety check silently produced no verdict for this PR, even though the model correctly determined there were no threats — the report path just wasn't propagated.

## Suggested fixes

1. Investigate why the tool-call result file (`/tmp/threat-detect-result-.json`) isn't found after the Copilot harness reports a successful `threat_detection_result` call — likely a race condition or incorrect file path resolution between the harness and `threat-detect`.
2. Consider surfacing a more visible warning/annotation when `no_verdict` occurs despite a successful engine run, distinguishing it from a genuine "model refused to report" case.
3. Consider retrying more than 0 times (`retries=0` in the log) before exhausting, or making retry count configurable per engine.

## Environment

- `gh-aw` version: v0.28.10 (`GH_AW_INFO_AWF_VERSION`)
- Engine: Copilot CLI `1.0.80` (`GH_AW_INFO_VERSION`)
- Runner: ubuntu-24.04, 20260823.283.1

Contributor guide

Open the contributing guide

Research direction

Start with the threat-detect and Copilot harness code paths that handle the successful threat_detection_result call and the temporary /tmp/threat-detect-result-.json path. Reproduce the linked workflow failure and trace how the tool-call output is written and read after the harness reports success. Done means a valid verdict is consumed without invalid_report_exhausted, and no-verdict failures are visibly distinguished from genuine model refusals.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go
Domain
ci-cd, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.