Detection job on arc-dind: threat-detect not staged, output path is read-only and unread
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 541
- Avg merge
- 5h 48m
- Merged PRs (30d)
- 773
Description
## Summary
With `runner: topology: arc-dind`, the `detection` job fails and threat detection never runs, while the workflow still concludes `success`. This is the same class of silent failure as #44249 (fixed by #44445 for the Copilot binary), but one binary later: `threat-detect` is never staged to a daemon-visible path.
The detection job is pinned to `runs-on: ubuntu-latest` and is not overridable from the frontmatter, yet it still receives the ARC/DinD codegen. There are three independent defects, listed below with line numbers from the generated lock.
## Environment
- gh-aw `v0.88.7`
- AWF `v0.28.14`
- threat-detect `v0.5.1`
- engine: `copilot`, CLI pinned to `1.0.79`
- runner: ARC (actions-runner-controller) with `containerMode: dind`, glibc runner image, Alpine `docker:28.2-dind` sidecar
- frontmatter: `runs-on: `, `runs-on-slim: `, `runner: topology: arc-dind`
## Symptom
The `detection` job reports two errors:
```
detection Process completed with exit code 127
detection ERR_SYSTEM: Detection result file not found at: /tmp/gh-aw/threat-detection/detection_result.json
detection Process completed with exit code 1
```
`conclude_threat_detection.sh` then reports:
```
RUN_DETECTION env: "true"
detection execution outcome: "failure"
Listing all files in artifact directory for diagnosis: /tmp/gh-aw/threat-detection
Found 16 file(s):
- agent_output.json
- detection.log
- sandbox/firewall/...
Detection log stats: 138 lines, 8612 bytes
No lines containing THREAT_DETECTION markers found in 138 lines
Threat Detection Engine Failure - The analysis engine could not complete.
```
The overall run still concludes `success`, so nothing surfaces except annotations.
## Defect 1: `threat-detect` is not staged to a daemon-visible path
The detection job installs the binary and then invokes it bare:
```yaml
# line 1874
- name: Install threat-detect binary
run: bash "${RUNNER_TEMP}/gh-aw/actions/install_threat_detect_binary.sh" v0.5.1
```
```bash
# line 1949
export PATH="${RUNNER_TEMP}/gh-aw/bin:$PATH" ... && threat-detect --engine copilot --output ...
```
The Copilot binary has an explicit staging step, added by #44445:
```yaml
# line 1868
- name: Copy Copilot CLI to daemon-visible path
run: |
mkdir -p "${RUNNER_TEMP}/gh-aw/bin"
COPILOT_SRC="$(command -v copilot)"
cp "$COPILOT_SRC" "${RUNNER_TEMP}/gh-aw/bin/copilot"
```
`threat-detect` has no equivalent. Without `arc-dind` the chroot is the daemon host filesystem, so `/usr/local/bin/threat-detect` resolves. With `arc-dind` the chroot is the staged sysroot, so it does not, which produces the `exit 127`.
## Defect 2: the output path is inside a read-only mount
```bash
# line 1947
awf --config ... \
--mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro" \
--mount "${RUNNER_TEMP}/gh-aw/home:${RUNNER_TEMP}/gh-aw/home:rw" \
--mount "${RUNNER_TEMP}/gh-aw/sandbox/agent:${RUNNER_TEMP}/gh-aw/sandbox/agent:rw" \
--mount /tmp/gh-aw/threat-detection:/tmp/gh-aw/threat-detection:rw
```
```bash
# line 1949
threat-detect --engine copilot --output ${RUNNER_TEMP}/gh-aw/threat-detection/detection_result.json ${RUNNER_TEMP}/gh-aw/threat-detection
```
`${RUNNER_TEMP}/gh-aw` is mounted `ro`, so even with the binary on `PATH` the write would fail. `/tmp/gh-aw/threat-detection` is the mount that is `rw`.
## Defect 3: producer and consumer disagree on the path
The result is written to `${RUNNER_TEMP}/gh-aw/threat-detection/detection_result.json` (line 1949) but read from `/tmp/gh-aw/threat-detection/detection_result.json`:
```yaml
# line 1975
- name: Upload threat detection artifact
path: /tmp/gh-aw/threat-detection/detection_result.json
# line 2003
- name: Conclude threat detection
run: bash ".../conclude_threat_detection.sh" /tmp/gh-aw/threat-detection/detection_result.json
```
The firewall logs are copied across explicitly, so the collection point is clearly `/tmp/gh-aw/threat-detection`:
```bash
# lines 1966-1968
mkdir -p /tmp/gh-aw/threat-detection/sandbox/firewall
if [ -d ${RUNNER_TEMP}/gh-aw/sandbox/firewall/logs ]; then ... cp -r ...; fi
```
Only `detection_result.json` is missing from that copy.
## Why the detection job gets ARC/DinD codegen at all
The detection job is emitted with `runs-on: ubuntu-latest` (not overridable from the frontmatter), yet it also gets:
```yaml
# line 1832
- name: Redirect tool cache and install paths for ARC/DinD
run: |
mkdir -p "${RUNNER_TEMP}/gh-aw/tool-cache"
echo "RUNNER_TOOL_CACHE=${RUNNER_TEMP}/gh-aw/tool-cache" >> "$GITHUB_ENV"
```
and `"runner":{"topology":"arc-dind"}` in its `awf-config.json` (line 1926). The same step appears at line 438 for the `agent` job, where it is correct.
So the topology is applied to a job that never runs on ARC. Either the detection job should not receive the ARC/DinD codegen at all, or it should receive it completely, including the binary staging that the agent job gets.
## Suggested fix
Whichever direction is preferred:
1. do not emit the ARC/DinD tool-cache redirection and `topology: arc-dind` for jobs pinned to `ubuntu-latest`, or
2. add a `Copy threat-detect to daemon-visible path` step mirroring line 1868, and point `--output` at `/tmp/gh-aw/threat-detection/detection_result.json` so producer and consumer agree.
Independently of the path work, a detection engine failure currently leaves the workflow green. Given that threat detection is a prompt-injection control, it would help if that surfaced more loudly than an annotation.
Related: #44249, #44445.
Contributor guide
Research direction
Start by locating the generator for the detection job and inspect install_threat_detect_binary.sh, conclude_threat_detection.sh, and the generated steps around lines 1832-2003. Compare the detection job with the agent setup around line 438, then reproduce ARC/DinD behavior and verify binary visibility, a writable /tmp/gh-aw/threat-detection output, matching producer and consumer paths, and visible failure reporting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, go, shell
- Domain
- ci-cd, devops, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100