github / github/gh-aw

awf-net hardcoded to 172.30.0.0/24 collides with OpenShift's default service CIDR, breaking all DNS resolution in Squid

Open
#59,880 1 comment 0 reactions 0 assignees View on GitHub
awf community
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 48m
Merged PRs (30d)
773

Description

## Summary

AWF creates `awf-net` with a hardcoded subnet of `172.30.0.0/24`. This is **OpenShift's default service CIDR** (`spec.serviceNetwork` on `network.config/cluster`). AWF additionally assigns `awf-squid` the static address `172.30.0.10`, which on a default OpenShift cluster is the ClusterIP of `dns-default` (CoreDNS) in the `openshift-dns` namespace.

The result is that on any OpenShift/OKD/ARO cluster running AWF on a self-hosted ARC runner, Squid is configured to send its DNS queries to its own interface address. Every lookup fails, and therefore every `CONNECT` fails.

There is currently no way to override the subnet — the Docker daemon's `--default-address-pool` is silently ignored because a Compose-declared subnet always wins.

## Environment

| | |
|---|---|
| Platform | Azure Red Hat OpenShift (ARO) |
| Service CIDR | `172.30.0.0/16` (OpenShift default) |
| CoreDNS ClusterIP | `172.30.0.10` (OpenShift default) |
| Runner | ARC `gha-runner-scale-set`, DinD sidecar |
| AWF topology | `arc-dind` |
| Engine | `copilot` |

## Symptoms

Every `CONNECT` through Squid returns 503 with `HIER_NONE`, meaning no next hop was ever selected — i.e. the hostname was never resolved:

```
1788971222.731 172.30.0.30:56126 api.githubcopilot.com:443 -:- 1.1 CONNECT 503
TCP_TUNNEL:HIER_NONE api.githubcopilot.com:443 "-"
1788971222.731 172.30.0.30:56118 api.githubcopilot.com:443 -:- 1.1 CONNECT 503
TCP_TUNNEL:HIER_NONE api.githubcopilot.com:443 "-"
```

Multiple parallel connections fail at an identical timestamp, which is Squid's negative DNS cache replaying a stored failure rather than retrying.

The Docker daemon is affected by the same collision:

```
docker: Error response from daemon: failed to resolve reference
"docker.io/library/alpine:latest": ... dial tcp: lookup registry-1.docker.io
on 172.30.0.10:53: read udp 172.30.0.1:60061->172.30.0.10:53: read: connection refused
```

Note the source address `172.30.0.1` — the `awf-net` bridge gateway. The query never left the bridge.

## Root cause

**1. Squid is configured to use `172.30.0.10` as its nameserver.**

From the `awf-squid` startup log:

```
2026/09/09 18:26:11| Adding nameserver 172.30.0.10 from squid.conf
```

The `from squid.conf` suffix confirms this is an explicit `dns_nameservers` directive, not an inherited `/etc/resolv.conf` value:

```
$ docker exec awf-squid grep -n dns_nameservers /etc/squid/squid.conf
176:dns_nameservers 172.30.0.10
```

This value appears to be templated from the runner pod's `/etc/resolv.conf`, which on OpenShift contains the CoreDNS ClusterIP.

**2. Squid itself is assigned that same address on `awf-net`.**

```
$ docker inspect awf-squid -f '{{json .NetworkSettings.Networks}}' | jq
{
"awf-...._awf-ext": {
"Gateway": "10.222.1.1",
"IPAddress": "10.222.1.2"
},
"awf-net": {
"IPAMConfig": { "IPv4Address": "172.30.0.10" },
"Gateway": "",
"IPAddress": "172.30.0.10"
}
}
```

**3. The on-link route wins.**

Once `awf-net` exists, `172.30.0.0/24` is an on-link route in the container's namespace. The kernel delivers the DNS query locally instead of routing it to the service proxy. Nothing listens on UDP/53 there, so the query is refused. Squid caches the failure and every subsequent `CONNECT` returns `HIER_NONE` 503.

The same shadowing affects the runner container and the dind daemon, since the bridge lives in the pod's network namespace.

## The subnet is not overridable

```
$ grep -rn "172\.30\." .../sandbox/firewall/audit/docker-compose.redacted.yml
398: - subnet: 172.30.0.0/24
```

Setting `--default-address-pool` on the Docker daemon does not help. It is correctly applied to networks Docker allocates itself, but a Compose-declared subnet takes precedence:

```
$ docker info | grep -A2 "Default Address Pool"
Default Address Pools:
Base: 10.222.0.0/16, Size: 24

$ docker network create __probe && docker network inspect __probe -f '{{json .IPAM.Config}}'
[{"Subnet":"10.222.1.0/24","Gateway":"10.222.1.1"}] # pool honoured

$ docker network inspect awf-net -f '{{json .IPAM.Config}}'
[{"Subnet":"172.30.0.0/24","Gateway":"172.30.0.1"}] # pool ignored
```

`awf-ext` picked up `10.222.1.0/24` from the pool correctly. Only `awf-net` is affected.

## Why this hasn't been reported before

Three conditions have to hold at once:

1. **Self-hosted runners.** On GitHub-hosted runners there is no service CIDR, so `172.30.0.0/24` is genuinely free.
2. **OpenShift specifically.** `172.30.0.0/16` is OpenShift's default service network. Vanilla Kubernetes defaults to `10.96.0.0/12`, AKS to `10.0.0.0/16`, EKS to `172.20.0.0/16` — none collide.
3. **The `arc-dind` topology.** Under the previous chroot mode, AWF pre-resolved allow-listed domains into `/etc/hosts`, so no live DNS lookups occurred and the broken resolver path was never exercised. `arc-dind` removes that step by design, which converts a latent address overlap into a hard failure.

## Requested change

The specific address is less important than the lack of an override. Suggestions in rough priority order:

1. **Make the subnet configurable** — a `--network-subnet` CLI flag (or equivalent config key) that flows into the generated Compose file, so operators on any platform can move `awf-net` out of the way.
2. **Detect the collision at startup and fail loudly.** Before creating the network, check the host/pod routing table and `/etc/resolv.conf` for addresses inside the proposed subnet. A clear error at setup time would be far cheaper to diagnose than `HIER_NONE` 503s several layers up.
3. **Do not template `dns_nameservers` from a resolver that falls inside `awf-net`.** Even with the subnet unchanged, refusing to write a nameserver address that AWF itself is about to claim would prevent the self-referential config.
4. **Consider defaulting to a less contested range.** `172.30.0.0/24` is a documented Kubernetes-distribution default; something in an unallocated part of RFC1918 would collide less often.

## Workaround

For anyone hitting this before a fix lands: point the runner pod at a resolver outside `172.30.0.0/16`, so AWF templates a reachable address into `squid.conf`.

OpenShift's CoreDNS uses `forward . /etc/resolv.conf`, so the node's own resolver is the correct target:

```bash
oc debug node/ -- chroot /host cat /etc/resolv.conf
```

Then in the ARC scale-set values (`containerMode` must be removed so `template.spec` is hand-written):

```yaml
template:
spec:
dnsPolicy: None
dnsConfig:
nameservers:
- ""
options:
- name: ndots
value: "1"
```

Squid is dual-homed, and its `awf-ext` leg carries the default route. Once the nameserver address no longer matches the on-link `awf-net` route, queries take the default route out and resolve normally.

Trade-off: this removes cluster DNS from the runner pod, so `*.svc.cluster.local` names no longer resolve. That is acceptable for AWF itself — the agent uses Docker's embedded resolver for sibling containers and Squid only resolves public domains — but it will break any workflow step that resolves an in-cluster Service.

Confirmed working: after this change, `dns_nameservers` is templated with the reachable address and `CONNECT` requests succeed with `TCP_TUNNEL:HIER_DIRECT/`.

Contributor guide

Open the contributing guide

Research direction

Trace the code that generates the Compose configuration, starting with the awf-net definition shown in docker-compose.redacted.yml and the CLI or configuration entry point for network settings. Reproduce the collision using the documented OpenShift service CIDR and verify how dns_nameservers is templated. Done means the subnet can be changed or the collision is rejected clearly before setup, with existing networking behavior still covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, go
Domain
devops, infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.