github / github/gh-aw

Detection job on arc-dind: threat-detect not staged, output path is read-only and unread

Open
#59,935 2 comments 0 reactions 0 assignees View on GitHub
awf community
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 48m
Merged PRs (30d)
773

Description

## Summary

With `runner: topology: arc-dind`, the `detection` job fails and threat detection never runs, while the workflow still concludes `success`. This is the same class of silent failure as #44249 (fixed by #44445 for the Copilot binary), but one binary later: `threat-detect` is never staged to a daemon-visible path.

The detection job is pinned to `runs-on: ubuntu-latest` and is not overridable from the frontmatter, yet it still receives the ARC/DinD codegen. There are three independent defects, listed below with line numbers from the generated lock.

## Environment

- gh-aw `v0.88.7`
- AWF `v0.28.14`
- threat-detect `v0.5.1`
- engine: `copilot`, CLI pinned to `1.0.79`
- runner: ARC (actions-runner-controller) with `containerMode: dind`, glibc runner image, Alpine `docker:28.2-dind` sidecar
- frontmatter: `runs-on: `, `runs-on-slim: `, `runner: topology: arc-dind`

## Symptom

The `detection` job reports two errors:

```
detection Process completed with exit code 127
detection ERR_SYSTEM: Detection result file not found at: /tmp/gh-aw/threat-detection/detection_result.json
detection Process completed with exit code 1
```

`conclude_threat_detection.sh` then reports:

```
RUN_DETECTION env: "true"
detection execution outcome: "failure"
Listing all files in artifact directory for diagnosis: /tmp/gh-aw/threat-detection
Found 16 file(s):
- agent_output.json
- detection.log
- sandbox/firewall/...
Detection log stats: 138 lines, 8612 bytes
No lines containing THREAT_DETECTION markers found in 138 lines
Threat Detection Engine Failure - The analysis engine could not complete.
```

The overall run still concludes `success`, so nothing surfaces except annotations.

## Defect 1: `threat-detect` is not staged to a daemon-visible path

The detection job installs the binary and then invokes it bare:

```yaml
# line 1874
- name: Install threat-detect binary
run: bash "${RUNNER_TEMP}/gh-aw/actions/install_threat_detect_binary.sh" v0.5.1
```

```bash
# line 1949
export PATH="${RUNNER_TEMP}/gh-aw/bin:$PATH" ... && threat-detect --engine copilot --output ...
```

The Copilot binary has an explicit staging step, added by #44445:

```yaml
# line 1868
- name: Copy Copilot CLI to daemon-visible path
run: |
mkdir -p "${RUNNER_TEMP}/gh-aw/bin"
COPILOT_SRC="$(command -v copilot)"
cp "$COPILOT_SRC" "${RUNNER_TEMP}/gh-aw/bin/copilot"
```

`threat-detect` has no equivalent. Without `arc-dind` the chroot is the daemon host filesystem, so `/usr/local/bin/threat-detect` resolves. With `arc-dind` the chroot is the staged sysroot, so it does not, which produces the `exit 127`.

## Defect 2: the output path is inside a read-only mount

```bash
# line 1947
awf --config ... \
--mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro" \
--mount "${RUNNER_TEMP}/gh-aw/home:${RUNNER_TEMP}/gh-aw/home:rw" \
--mount "${RUNNER_TEMP}/gh-aw/sandbox/agent:${RUNNER_TEMP}/gh-aw/sandbox/agent:rw" \
--mount /tmp/gh-aw/threat-detection:/tmp/gh-aw/threat-detection:rw
```

```bash
# line 1949
threat-detect --engine copilot --output ${RUNNER_TEMP}/gh-aw/threat-detection/detection_result.json ${RUNNER_TEMP}/gh-aw/threat-detection
```

`${RUNNER_TEMP}/gh-aw` is mounted `ro`, so even with the binary on `PATH` the write would fail. `/tmp/gh-aw/threat-detection` is the mount that is `rw`.

## Defect 3: producer and consumer disagree on the path

The result is written to `${RUNNER_TEMP}/gh-aw/threat-detection/detection_result.json` (line 1949) but read from `/tmp/gh-aw/threat-detection/detection_result.json`:

```yaml
# line 1975
- name: Upload threat detection artifact
path: /tmp/gh-aw/threat-detection/detection_result.json

# line 2003
- name: Conclude threat detection
run: bash ".../conclude_threat_detection.sh" /tmp/gh-aw/threat-detection/detection_result.json
```

The firewall logs are copied across explicitly, so the collection point is clearly `/tmp/gh-aw/threat-detection`:

```bash
# lines 1966-1968
mkdir -p /tmp/gh-aw/threat-detection/sandbox/firewall
if [ -d ${RUNNER_TEMP}/gh-aw/sandbox/firewall/logs ]; then ... cp -r ...; fi
```

Only `detection_result.json` is missing from that copy.

## Why the detection job gets ARC/DinD codegen at all

The detection job is emitted with `runs-on: ubuntu-latest` (not overridable from the frontmatter), yet it also gets:

```yaml
# line 1832
- name: Redirect tool cache and install paths for ARC/DinD
run: |
mkdir -p "${RUNNER_TEMP}/gh-aw/tool-cache"
echo "RUNNER_TOOL_CACHE=${RUNNER_TEMP}/gh-aw/tool-cache" >> "$GITHUB_ENV"
```

and `"runner":{"topology":"arc-dind"}` in its `awf-config.json` (line 1926). The same step appears at line 438 for the `agent` job, where it is correct.

So the topology is applied to a job that never runs on ARC. Either the detection job should not receive the ARC/DinD codegen at all, or it should receive it completely, including the binary staging that the agent job gets.

## Suggested fix

Whichever direction is preferred:

1. do not emit the ARC/DinD tool-cache redirection and `topology: arc-dind` for jobs pinned to `ubuntu-latest`, or
2. add a `Copy threat-detect to daemon-visible path` step mirroring line 1868, and point `--output` at `/tmp/gh-aw/threat-detection/detection_result.json` so producer and consumer agree.

Independently of the path work, a detection engine failure currently leaves the workflow green. Given that threat detection is a prompt-injection control, it would help if that surfaced more loudly than an annotation.

Related: #44249, #44445.

Contributor guide

Open the contributing guide

Research direction

Start by locating the generator for the detection job and inspect install_threat_detect_binary.sh, conclude_threat_detection.sh, and the generated steps around lines 1832-2003. Compare the detection job with the agent setup around line 438, then reproduce ARC/DinD behavior and verify binary visibility, a writable /tmp/gh-aw/threat-detection output, matching producer and consumer paths, and visible failure reporting.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, go, shell
Domain
ci-cd, devops, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.