Azure / Azure/unbounded

Flake: agent config e2e Kindnet uses unreachable control-plane Node IP

Open
#467 0 comments 0 reactions 0 assignees View on GitHub
bug github_actions node agent
Dominant language
Go
Stars
28
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
55

Description

## Summary

The `agent config e2e` job can fail because Kindnet on QEMU/nspawn worker nodes attempts to route the control-plane PodCIDR through the Kind container's Docker address, which is not reachable from the VM network.

Observed in PR #464:

https://github.com/Azure/unbounded/actions/runs/29282078424/job/86925475493

## Failure

The worker's Kindnet pod repeatedly panics:

```text
Adding route ... Dst: 10.244.0.0/24 Gw: 172.18.0.2
Failed to reconcile routes, retrying after error: network is unreachable
panic: Maximum retries reconciling node routes: network is unreachable
```

The worker only has connectivity to the `192.168.110.0/24` VM bridge. The Kind control-plane container is attached to that bridge as `192.168.110.2`, but its Kubernetes Node continues advertising the Docker address `172.18.0.2`.

Kindnet therefore selects an unreachable next hop for the control-plane PodCIDR. Containerd never receives a usable CNI configuration, leaving the worker:

```text
NetworkPluginNotReady: cni plugin not initialized
```

The test currently deletes crashing Kindnet pods every 30 seconds. This occasionally succeeds by timing, but does not repair the invalid route. A previous successful run required approximately 680 seconds and many restarts.

## Proposed fix

Make the Kind control-plane Node advertise its VM-bridge address before any QEMU worker joins.

1. Create `virbr-e2e` and attach the Kind container using `eth-e2e` at `${VM_SUBNET}.2`.
2. Update `/var/lib/kubelet/kubeadm-flags.env` inside the Kind container:
- remove any existing `--node-ip`;
- append `--node-ip=${VM_SUBNET}.2`.
3. Restart the Kind control-plane kubelet.
4. Wait until the control-plane Node reports `${VM_SUBNET}.2` as its InternalIP.
5. Patch/restart Kindnet only after this topology is established.
6. Keep `CONTROL_PLANE_ENDPOINT` configured separately for API access.
7. Apply the same setup to CI and `hack/agent/e2e-kind/run-local.sh`.

`hack/smoke-metalman.py:430-469` already implements equivalent kubelet Node-IP configuration and can be reused or factored into a shared helper.

## Likely files

- `.github/actions/agent-e2e-kind-control-plane/action.yaml`
- `hack/agent/e2e-kind/e2e.py`
- `hack/agent/e2e-kind/run-local.sh`

## Acceptance criteria

- The Kind control-plane Node advertises `${VM_SUBNET}.2`.
- Worker Kindnet routes the control-plane PodCIDR through `${VM_SUBNET}.2`, not `172.18.x.x`.
- Kindnet does not panic with `network is unreachable`.
- Worker nodes become Ready without repeated Kindnet pod deletion.
- `agent config e2e` passes consistently across multiple consecutive runs.
- CI and local setup use the same network topology.

Contributor guide

Open the contributing guide

Research direction

Read .github/actions/agent-e2e-kind-control-plane/action.yaml, hack/agent/e2e-kind/e2e.py, and hack/agent/e2e-kind/run-local.sh, then compare hack/smoke-metalman.py:430-469. Run the agent config e2e setup and verify the control-plane Node reports ${VM_SUBNET}.2 before workers join. Done means consistent CI and local passes, usable worker routes, Ready workers, and no Kindnet network-unreachable panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, kubernetes, python, shell
Domain
ci-cd, devops, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.