Sysbox v0.6.1 daemon-set installer fails in GKE 1.24.10-gke.2300 with DataplaneV2 (Cilium) or Calico
@rodnymolina is already working on this.
Since Apr 20, 2023.
- Dominant language
- Shell
- Stars
- 3.9k
- Forks
- 230
- Avg merge
- 7h 48m
- Merged PRs (30d)
- 3
Description
It appears as though during the sysbox install process (via your daemonset) I get some extra cri-o configs 100-crio-bridge.conf, 200-loopback.conf co-located in /etc/cni/net.d that breaks my GKE networking and introduces a subnet for the node that's not correct according to what's been configured in GKE by the 10-gke-ptp.conflist config file already there.
When pods come back online after the kubelet restart on that node, a number of them including the sysbox-installer get this incorrect CIDR provided in 100-crio-bridge.conf and they go into crash loop backoff state as they cannot reach anything on the network (which is correct, there are no firewall/policy rules allowing traffic from this new CIDR that came out of thin air.)
For reference, here's the content of 100-crio-bridge.conf that gets placed into /etc/cni/net.d and breaks the networking for pods running on the node I'm trying to install sysbox on:
{
"cniVersion": "0.3.1",
"name": "crio",
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"hairpinMode": true,
"ipam": {
"type": "host-local",
"routes": [
{ "dst": "0.0.0.0/0" },
{ "dst": "1100:200::1/24" }
],
"ranges": [
[{ "subnet": "10.85.0.0/16" }],
[{ "subnet": "1100:200::/24" }]
]
}
}
I've tried restarting various services on the node, along with just restarting the node entirely after deleting these extraneous files from /etc/cni/net.d (while ensuring I keep 10-gke-ptp.conflist) but no dice, I just get a ton of messages like this:
Apr 19 21:27:34 gha-std-runners-2-baae0455-8zr5 kubelet[25451]: E0419 21:27:34.993101 25451 pod_workers.go:965] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/cni/net.d/. Has your network provider started?" pod="kube-system/sysbox-deploy-k8s-98fpk" podUID=3287c25f-d5f2-477e-a934-918722a0908a
@rodnymolina for viz as you asked me to bring this in as an Issue to track.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.