github / github/gh-aw

threat-detect: "compact fallback prompt" returns 400 Bad Request when the full prompt exceeds 100KB

Open
#59,038 0 comments 0 reactions 1 assignee Claimed by @davidslater View on GitHub
community threat-detection
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 48m
Merged PRs (30d)
773

Description

### Summary

When the threat-detection prompt built for a PR review exceeds 100KB, `threat-detect` (v0.5.1) switches to a "compact fallback prompt" instead of the full one. That fallback path fails immediately with a `400 Bad Request` from the Copilot backend, with zero retries configured, so threat detection reports `agent_failure` on every run where the diff is large enough to trigger the fallback — even though the underlying PR review itself completes successfully.

### Environment

- gh-aw compiler: v0.88.4
- threat-detect: v0.5.1
- Engine: copilot, model `gpt-5.4-mini`
- A PR-review workflow with native (compiler-default) threat detection, `continue-on-error: true` (default)

### Observed behavior

Two consecutive runs on the same PR (large diff, big codebase) both failed identically:

```
[copilot-harness] resolved --prompt-file: path=/tmp/threat-detect-prompt-XXXXXXXXXX.txt size=118029B
[copilot-harness] prompt file exceeds 100KB; using compact fallback prompt
[copilot-harness] attempt 1: spawning: /usr/local/bin/copilot --add-dir /tmp --log-level all --disable-builtin-mcps --no-ask-user --allow-all-tools --add-dir -p
[copilot-harness] attempt 1: process started (pid=184)

400 400 400 Bad Request

Changes +0 -0
Duration 0s
Resume copilot --resume=
[copilot-harness] attempt 1: process exit event exitCode=1
[copilot-harness] attempt 1: process closed exitCode=1 duration=1s stdout=0B stderr=123B hasOutput=true
[copilot-harness] attempt 1 failed: exitCode=1 failureClass=partial_execution ... tokenCount=0 attemptDurationMs=1657 retriesRemaining=0
[copilot-harness] all 0 retries exhausted — giving up (exitCode=1)
Error running detection: engine execution failed: node exited with code 1: ...
THREAT_DETECTION_STATUS: reason=engine_error exit=2
##[error]ERR_SYSTEM: ❌ Detection result file not found at: /tmp/gh-aw/threat-detection/detection_result.json
```

The failure is immediate (attemptDurationMs≈1657, duration=1s) and 100% reproducible on this PR — both runs (7 minutes apart, one after a fresh commit) hit the identical `400 Bad Request` at the identical stage, right after the "using compact fallback prompt" line.

### Impact

- `safe_outputs`, `conclusion`, and the main PR review comment all complete successfully (`continue-on-error: true` on the detection job absorbs this).
- The gh-aw-managed "Detection Runs" tracking issue accumulates a `failure` / `agent_failure` comment on every affected run, with no way to distinguish "genuine security concern" from "tooling crashed" other than reading the run log.
- Any repo whose typical PR diffs regularly exceed 100KB (large codebases, generated/lock files, vendored code) will see this on effectively every PR review.

### Suspected root cause

The "compact fallback prompt" that gets substituted once the full prompt exceeds 100KB appears to produce a request the Copilot backend rejects outright (400, zero tokens, sub-2-second failure) — as opposed to the full prompt, which presumably works (no 100KB-related failures are seen on smaller PRs). This suggests the fallback-prompt construction path itself is malformed (e.g. missing a required field, invalid encoding, or truncating mid-structure) rather than the request being legitimately too large for the model.

### Suggested fix direction

- Validate/test the compact-fallback-prompt code path explicitly (it looks under-exercised, since it seems to always fail rather than degrade gracefully).
- Alternatively, raise the compaction threshold, or compact more conservatively (e.g. truncate the diff content rather than whatever is currently dropped) so the resulting request is still well-formed.
- At minimum, retry the compact-fallback path at least once before giving up (currently `retriesRemaining=0` on the very first attempt), since this looks like a request-construction bug rather than a transient error.

### Reproduction notes

Happy to share the full job log / the exact >100KB prompt that triggers this if that's useful, but the pattern above should be reproducible on any sufficiently large PR diff with threat-detection enabled.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.