github / github/gh-aw

External threat detection fails on ARC/DinD: `threat-detect` is not staged into the chroot

Open
#59,487 0 comments 1 reaction 2 assignees Claimed by @pelikhan View on GitHub
ai-inspected community threat-detection
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 48m
Merged PRs (30d)
773

Description

## Summary

The default external threat detector installs successfully on the runner, but the generated detection job does not make `threat-detect` available inside the AWF chroot on `runner.topology: arc-dind`. Bash then exits 127 before the detector starts.

This is a gh-aw-generated job issue, not a request to change the runner image. Node is already available inside the chroot in the affected runs, so this is distinct from github/gh-aw#59235.

Observed on two internal workflow runs with gh-aw v0.88.2, AWF v0.28.12, Copilot CLI 1.0.80 and threat-detect v0.5.1. The same compiler omission remains in the published v0.88.3-v0.88.6 sources. A separate input/output path mismatch is also present in the generated job, but its runtime effect has not been exercised because the binary never starts.

With the default `continue-on-error: true`, both affected detection jobs and overall runs concluded `success`, while detection reported `failure` / `agent_failure`. This is an infrastructure failure, not an observed threat verdict. The configured warn-mode policy is not the bug being reported.

## Requested fix: agentic implementation plan

Please implement this in `pkg/workflow` and the existing setup scripts:

1. Add failing regression coverage in `threat_detection_steps_test.go`, `threat_detection_external_detector_execution_test.go`, and `threat_detection_job_combinations_integration_test.go` for external detection on ARC/DinD.
2. Update `buildInstallThreatDetectStep` to respect the detection job's rootless installation policy and stage the verified binary under `${RUNNER_TEMP}/gh-aw/bin`. Make it resolvable inside AWF independently of Copilot-specific PATH setup; retain runner-side availability for conclusion.
3. Make preparation, sandbox arguments, writable mounts, result collection, upload and conclusion agree on detection paths. Reuse `prepare_threat_detection_files.sh` and its destination argument. Keep the outer runtime mount read-only and grant write access only to the detection subdirectory.
4. Preserve the non-ARC and inline paths, authentication isolation, result/output contracts, log-exclusion policy, and both `continue-on-error` modes. Cover skipped detection and failed installation/staging without manufacturing a successful verdict.
5. Extend the existing installer/preparation shell tests, update `docs/src/content/docs/reference/threat-detection.md`, and run `make agent-finish` in the supported Dev Container/Codespace.

**Acceptance:** an ARC/DinD external detection run with Node available and benign, non-empty safe output starts the detector and produces a valid result consumed by the existing conclusion step, without command-not-found, missing-input, read-only-output or missing-result errors.

## Runtime evidence

The runs were inspected with `gh run view --job ... --log` and detection artifacts downloaded with `gh aw audit`. The excerpts below omit timestamps, decorative formatting and private repository identifiers; they do not require access to the internal repository.

Both runs show this sequence:

```text
Installing threat-detect with checksum verification (version: v0.5.1, os: Linux, arch: x86_64)
threat-detect v0.5.1
threat-detect installation complete
[entrypoint] Runner binaries overlay detected at /tmp/awf-runner-bin; prepended to PATH
[entrypoint] Node.js available via GH_AW_NODE_BIN: /home/runner/_work/_temp/gh-aw/tool-cache/node/bin/node
/bin/bash: line 2: threat-detect: command not found
[WARN] Command completed with exit code: 127
```

The conclusion step subsequently reports:

```text
RUN_DETECTION: true
ERR_SYSTEM: Detection result file not found at: /tmp/gh-aw/threat-detection/detection_result.json
```

The generated steps install AWF with `--rootless` and copy Copilot into `${RUNNER_TEMP}/gh-aw/bin/copilot`, but install `threat-detect` without `--rootless` and emit no equivalent copy for it:

```yaml
- name: Install AWF binary
run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.28.12 --rootless
# Copilot installation and other setup steps omitted.
- name: Copy Copilot CLI to daemon-visible path
run: |
mkdir -p "${RUNNER_TEMP}/gh-aw/bin"
COPILOT_SRC="$(command -v copilot)"
cp "$COPILOT_SRC" "${RUNNER_TEMP}/gh-aw/bin/copilot"
chmod +x "${RUNNER_TEMP}/gh-aw/bin/copilot"
- name: Install threat-detect binary
if: always() && steps.detection_guard.outputs.run_detection == 'true'
continue-on-error: true
run: |
bash "${RUNNER_TEMP}/gh-aw/actions/install_threat_detect_binary.sh" v0.5.1
```

Source analysis and release scope

### Missing binary staging: observed startup failure

- [`buildInstallThreatDetectStep`, v0.88.6](https://github.com/github/gh-aw/blob/v0.88.6/pkg/workflow/threat_detection_steps.go#L580-L599) passes only the version to the installer.
- [`install_threat_detect_binary.sh`](https://github.com/github/gh-aw/blob/v0.88.6/actions/setup/sh/install_threat_detect_binary.sh#L33-L63) selects `/usr/local/bin` unless `--rootless` is supplied. Installation and the runner-side version check succeed in these runs.
- [`buildExternalDetectorPathSetup`](https://github.com/github/gh-aw/blob/v0.88.6/pkg/workflow/threat_detection_external.go#L208-L237) returns the Copilot PATH prefix on ARC/DinD without a host setup command. The existing Copilot copy step does not stage the separate detector binary.
- [`buildInstallAWFForExternalDetectorStep`](https://github.com/github/gh-aw/blob/v0.88.6/pkg/workflow/threat_detection_external.go#L239-L261) already derives installation mode from the detection sandbox configuration; the detector installer should follow the applicable policy too.

The runtime Compose artifact confirms a build-tools-populated `sysroot:/host:rw` volume, with the runner's shared `${RUNNER_TEMP}/gh-aw` exposed both at its absolute path and at `/host/tmp/awf-runner-bin`. This is a separate chroot filesystem, not simply the runner's `/usr/local/bin`, nor literally the DinD daemon's root filesystem.

AWF's [system mount builder](https://github.com/github/gh-aw-firewall/blob/v0.28.14/src/services/agent-volumes/system-mounts.ts#L15-L42) exposes the configured binaries directory, and the [entrypoint](https://github.com/github/gh-aw-firewall/blob/v0.28.14/containers/agent/entrypoint.sh#L780-L820) adds that overlay to PATH. Neither operation copies an unstaged runner-local `threat-detect` into it.

**Adding `--rootless` alone is insufficient:** changing installation to runner-local `~/.local/bin` does not itself make the binary visible inside the chroot.

### Path mismatch: code/configuration evidence, not a second runtime reproduction

The external command is built with `/tmp/gh-aw/threat-detection` in [`threat_detection_external.go`](https://github.com/github/gh-aw/blob/v0.88.6/pkg/workflow/threat_detection_external.go#L410-L444), then [ARC command rewriting](https://github.com/github/gh-aw/blob/v0.88.6/pkg/workflow/awf_arc_dind.go#L15-L22) changes those paths to `${RUNNER_TEMP}/gh-aw/threat-detection`.

Meanwhile:

- [Preparation](https://github.com/github/gh-aw/blob/v0.88.6/actions/setup/sh/prepare_threat_detection_files.sh#L6-L20) defaults to `/tmp/gh-aw/threat-detection`.
- The explicit [writable detection mount](https://github.com/github/gh-aw/blob/v0.88.6/pkg/workflow/threat_detection_external.go#L302-L308) still uses `/tmp/gh-aw/threat-detection`.
- Artifact upload and conclusion also use the original `/tmp/gh-aw/threat-detection/detection_result.json`.

Relevant generated arguments, also present in the runtime Compose artifact:

```bash
--mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro"
--mount /tmp/gh-aw/threat-detection:/tmp/gh-aw/threat-detection:rw

threat-detect --engine copilot \
--output ${RUNNER_TEMP}/gh-aw/threat-detection/detection_result.json \
${RUNNER_TEMP}/gh-aw/threat-detection
```

These are excerpts, not a complete shell command. The rewritten input directory has no corresponding preparation step, and the rewritten output lies under the read-only runtime mount. This must be covered when fixing startup, rather than claiming that copying the binary alone restores detection.

A concrete layout to validate is to prepare the full input directory under `${RUNNER_TEMP}/gh-aw/threat-detection`, mount just that directory read-write, and collect the result back to the existing `/tmp/gh-aw/threat-detection` location before upload/conclusion. Missing or invalid results must retain the existing failure handling. Do not rewrite all agent paths globally or broaden the runtime mount to read-write.

### Published versions inspected on 2026-09-08

| gh-aw | AWF default | Detector | Evidence |
|---|---|---|---|
| v0.88.2 | v0.28.12 | v0.5.1 | Two runtime failures and source inspection |
| v0.88.3 | v0.28.13 | v0.5.1 | Source inspection only |
| v0.88.4 | v0.28.13 | v0.5.1 | Source inspection only |
| v0.88.5 | v0.28.13 | v0.5.1 | Source inspection only |
| v0.88.6 | v0.28.14 | v0.5.1 | Source inspection only |

Across these five tags, `threat_detection_external.go`, `threat_detection_steps.go`, `threat_detection_job.go`, `awf_arc_dind.go`, and the detector installer are individually byte-identical. For example, the external-detector file has Git blob SHA `478063ee7c6ac00605f9b9983ac5d32e1ddf65a9` throughout.

The bundled AWF upgrades were also inspected: the binary staging, system mount, volume builder, sysroot service and build-tools Dockerfile are unchanged across v0.28.12-v0.28.14. The changed browser-library staging, working-directory mounts and credential-overlay handling do not expose this runner-installed detector.

This is evidence that the relevant omissions remain in the released sources, not a claim of runtime reproduction on every prerelease.

Minimal reproduction workflow and execution prerequisites

Use a Linux ARC/DinD runner with a shared workspace/runtime directory visible to both runner and daemon. Replace `arc-dind-runner` in all four runner fields. The repository must support Copilot inference through `copilot-requests: write`.

Save the following as `.github/workflows/repro-external-detection-arc.md`. It deliberately provisions and stages Node to isolate this failure from github/gh-aw#59235; this is reproduction scaffolding, not a proposed permanent runner workaround.

```markdown
---
on:
workflow_dispatch:
runs-on: arc-dind-runner
runs-on-slim: arc-dind-runner
runner:
topology: arc-dind
permissions:
contents: read
copilot-requests: write
engine:
id: copilot
version: "1.0.80"
network: defaults
timeout-minutes: 10
jobs:
detection:
pre-steps:
- name: Provision Node for detection reproduction
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "24"
package-manager-cache: false
- name: Stage Node for detection reproduction
shell: bash
run: |
set -euo pipefail
NODE_BIN="$(command -v node)"
NODE_PREFIX="$(dirname "$(dirname "$NODE_BIN")")"
NODE_DEST="${RUNNER_TEMP}/gh-aw/tool-cache/detection-node"
mkdir -p "$NODE_DEST"
cp -a "$NODE_PREFIX"/. "$NODE_DEST"/
test -x "$NODE_DEST/bin/node"
echo "$NODE_DEST/bin" >> "$GITHUB_PATH"
echo "GH_AW_NODE_BIN=$NODE_DEST/bin/node" >> "$GITHUB_ENV"
safe-outputs:
runs-on: arc-dind-runner
threat-detection:
runs-on: arc-dind-runner
continue-on-error: false
create-issue:
max: 1
---

Call the create_issue safe-output tool once with title
"External detector ARC reproduction" and body
"Benign output to exercise threat detection." Do not use noop.
```

Compile with gh-aw v0.88.2:

```bash
gh aw compile repro-external-detection-arc
```

Commit the source and generated lock file to a test repository and dispatch the workflow from that revision. Ensure the agent produces a non-empty safe output and `RUN_DETECTION=true`.

The expected pre-fix signature is a successful Node preflight followed by `threat-detect: command not found`, exit 127, and a missing result. The example intentionally uses `continue-on-error: false` so safe-output processing is blocked on failure; the original observations used the default `true`. Once fixed, this example can create one test issue.

This exact reduced example compiled successfully with gh-aw v0.88.2. It is separate from the two observed production runs and has not been dispatched as a new ARC runtime reproduction.

Regression coverage and completion criteria

Use the existing test suites rather than adding a new runner:

- `pkg/workflow/threat_detection_steps_test.go`: installer invocation, install/stage/execute/collect/conclude ordering, and unchanged guard conditions.
- `pkg/workflow/threat_detection_external_detector_execution_test.go`: extend `TestBuildExternalDetectorPathSetup` and topology coverage to assert detector visibility, not just Copilot visibility.
- `pkg/workflow/threat_detection_job_combinations_integration_test.go`: external and inline paths, ARC and non-ARC, Copilot and other supported detector engines. Detector staging must not depend on whether the engine uses a standard or custom Copilot command.
- `actions/setup/sh/install_threat_detect_binary_test.sh`: retain checksum verification and cover rootless placement, executable permissions, failure propagation and runner-side availability.
- `actions/setup/sh/prepare_threat_detection_files_test.sh`: preserve all prepared inputs when using the shared destination, including prompt context, safe outputs and patches.
- `pkg/workflow/threat_detection_conclude_script_test.go`: skipped detection, successful results, missing results and both failure-policy modes.

In addition to compiler assertions, exercise the filesystem boundary on ARC/DinD: the runner's successful `threat-detect --version` is not proof that the sandbox can invoke it.

- The sandbox resolves and executes the staged detector.
- Prepared inputs are readable and the result destination is writable.
- A benign, non-empty input produces a valid structured result that upload and conclusion can find.
- Infrastructure failure produces no fabricated successful verdict and follows the configured policy.
- Raw `detection.log` remains excluded from the external detector artifact.
- Node availability remains intact, including when combined with the Node provisioning fix.
- Unrelated workflows retain their existing mounts, output contracts and failure defaults.

Run targeted regressions first, then `make agent-finish` in the project's Dev Container/Codespace. This issue proposes work for a core-team coding agent; it does not require an external-contributor PR.

## Related issues and temporary options

- github/gh-aw#59235: external detection omits Node provisioning. These runs pass Node preflight and fail at the next executable. Its fix, github/gh-aw#59314, merged on 2026-09-08, after v0.88.6 was published; Node provisioning and detector staging are separate requirements.
- github/gh-aw#44249: analogous missing Copilot staging in the older inline detector. The specific missing executable here is `threat-detect`, not Copilot.
- github/gh-aw#47016: missing detector invocation during skipped conclusion. Here detection is enabled and the failure happens inside AWF execution.

Setting `safe-outputs.threat-detection.continue-on-error: false` blocks safe outputs on detection failure; it does not repair startup.

`features.gh-aw-detection: false` selects the legacy inline detector rather than disabling detection. Its compilation was inspected, but it has not been demonstrated to resolve these runs end-to-end. Node must still be reachable, and the inline path uploads a potentially sensitive `detection.log` artifact that the external path deliberately excludes. It should not be presented as a risk-free or already-proven fix.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.