Flake: agent config e2e Kindnet uses unreachable control-plane Node IP
- Dominant language
- Go
- Stars
- 28
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 55
Description
## Summary
The `agent config e2e` job can fail because Kindnet on QEMU/nspawn worker nodes attempts to route the control-plane PodCIDR through the Kind container's Docker address, which is not reachable from the VM network.
Observed in PR #464:
https://github.com/Azure/unbounded/actions/runs/29282078424/job/86925475493
## Failure
The worker's Kindnet pod repeatedly panics:
```text
Adding route ... Dst: 10.244.0.0/24 Gw: 172.18.0.2
Failed to reconcile routes, retrying after error: network is unreachable
panic: Maximum retries reconciling node routes: network is unreachable
```
The worker only has connectivity to the `192.168.110.0/24` VM bridge. The Kind control-plane container is attached to that bridge as `192.168.110.2`, but its Kubernetes Node continues advertising the Docker address `172.18.0.2`.
Kindnet therefore selects an unreachable next hop for the control-plane PodCIDR. Containerd never receives a usable CNI configuration, leaving the worker:
```text
NetworkPluginNotReady: cni plugin not initialized
```
The test currently deletes crashing Kindnet pods every 30 seconds. This occasionally succeeds by timing, but does not repair the invalid route. A previous successful run required approximately 680 seconds and many restarts.
## Proposed fix
Make the Kind control-plane Node advertise its VM-bridge address before any QEMU worker joins.
1. Create `virbr-e2e` and attach the Kind container using `eth-e2e` at `${VM_SUBNET}.2`.
2. Update `/var/lib/kubelet/kubeadm-flags.env` inside the Kind container:
- remove any existing `--node-ip`;
- append `--node-ip=${VM_SUBNET}.2`.
3. Restart the Kind control-plane kubelet.
4. Wait until the control-plane Node reports `${VM_SUBNET}.2` as its InternalIP.
5. Patch/restart Kindnet only after this topology is established.
6. Keep `CONTROL_PLANE_ENDPOINT` configured separately for API access.
7. Apply the same setup to CI and `hack/agent/e2e-kind/run-local.sh`.
`hack/smoke-metalman.py:430-469` already implements equivalent kubelet Node-IP configuration and can be reused or factored into a shared helper.
## Likely files
- `.github/actions/agent-e2e-kind-control-plane/action.yaml`
- `hack/agent/e2e-kind/e2e.py`
- `hack/agent/e2e-kind/run-local.sh`
## Acceptance criteria
- The Kind control-plane Node advertises `${VM_SUBNET}.2`.
- Worker Kindnet routes the control-plane PodCIDR through `${VM_SUBNET}.2`, not `172.18.x.x`.
- Kindnet does not panic with `network is unreachable`.
- Worker nodes become Ready without repeated Kindnet pod deletion.
- `agent config e2e` passes consistently across multiple consecutive runs.
- CI and local setup use the same network topology.
Contributor guide
Research direction
Read .github/actions/agent-e2e-kind-control-plane/action.yaml, hack/agent/e2e-kind/e2e.py, and hack/agent/e2e-kind/run-local.sh, then compare hack/smoke-metalman.py:430-469. Run the agent config e2e setup and verify the control-plane Node reports ${VM_SUBNET}.2 before workers join. Done means consistent CI and local passes, usable worker routes, Ready workers, and no Kindnet network-unreachable panic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, kubernetes, python, shell
- Domain
- ci-cd, devops, infrastructure, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100