github / github/gh-aw

arc-dind: hardcoded CAP_SYS_MODULE in cap_drop breaks AWF on hosts with a trimmed capability bounding set (Talos)

Open
#56,127 1 comment 0 reactions 0 assignees View on GitHub
awf community
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 46m
Merged PRs (30d)
760

Description

### What happened

`runner.topology: arc-dind` gets a Copilot workflow almost all the way through on an ARC runner, but the run dies when AWF brings up its sandbox:

```
Error response from daemon: invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"
[ERROR] Fatal error: Error: Command failed with exit code 1: docker compose up -d --no-deps --pull never squid-proxy
```

The nodes run [Talos Linux](https://www.talos.dev/), which removes `CAP_SYS_MODULE` and `CAP_SYS_BOOT` from the bounding set of every container on the host — kernel module loading is not part of its threat model:

```
CapBnd: 000001ffffbeffff # Talos
CapBnd: 000001ffffffffff # stock
```

A privileged DinD sidecar inherits that bounding set and cannot exceed it, so the daemon has no `CAP_SYS_MODULE` to reason about. Docker validates `cap_drop` entries against the capabilities it can see and rejects the whole container create rather than treating the drop as satisfied.

### Where it comes from

`awf-bundle.js` hardcodes the list for both the squid proxy and the agent container:

```js
cap_drop:["NET_RAW","SYS_ADMIN","SYS_PTRACE","SYS_MODULE","MKNOD","AUDIT_WRITE","SETFCAP"]
```

```js
cap_add:["SYS_CHROOT","SYS_ADMIN"],cap_drop:["NET_RAW","SYS_PTRACE","SYS_MODULE","SYS_RAWIO","MKNOD"]
```

Still present in `v0.28.8`. I could not find an env var or config key that influences it — `buildContainerSecurityHardening` builds the list unconditionally.

### Why this is safe to change

Dropping a capability that is already outside the bounding set is a no-op. On a host like this, `SYS_MODULE` cannot be held by any container whether AWF names it or not, so omitting it costs exactly zero isolation. The request is only that AWF not *insist* on naming it.

### Suggested fix

Filter `cap_drop` against what the daemon reports before writing the compose file — `docker info` exposes the host's capability view, and `/proc/self/status` `CapBnd` is readable in the runner. Anything not present is already dropped. A narrower alternative would be a documented escape hatch such as `AWF_SKIP_CAP_DROP`, though auto-detection seems strictly better since it needs no configuration and cannot be used to weaken a host that *does* have the capability.

### Impact

This affects any hardened or minimal host that trims the bounding set, not just Talos — the same reasoning applies to distributions that drop `SYS_MODULE` for immutability. `arc-dind` is documented as a supported topology, and everything else in that path now works correctly; this is the only remaining blocker.

### Workaround in use

A `pre-agent-steps` step that strips `"SYS_MODULE"` from the installed bundle after `install_awf_binary.sh` runs, asserting that every other drop survives. Happy to withdraw it the moment this is handled upstream.

### Environment

| | |
|---|---|
| gh-aw | `v0.85.4` |
| AWF | `v0.27.44` (reproduced against `v0.28.8`) |
| Topology | `runner.topology: arc-dind`, ARC `gha-runner-scale-set` 0.14.2 |
| Host | Talos Linux, kernel `6.18.36-talos`, Kubernetes 1.36 |
| Sidecar | `docker:29-dind`, privileged, `DOCKER_HOST=tcp://localhost:2376` |

Contributor guide

Open the contributing guide

Research direction

Start with awf-bundle.js and buildContainerSecurityHardening, then inspect how the cap_drop lists are assembled before docker compose up runs. Reproduce the arc-dind path on the reported Talos environment and verify that containers start there while the complete capability drops remain on hosts that expose them.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, javascript, kubernetes, linux
Domain
devops, infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.