nestybox / nestybox/sysbox

Sysbox v0.6.1 daemon-set installer fails in GKE 1.24.10-gke.2300 with DataplaneV2 (Cilium) or Calico

Open
#680 9 comments 0 reactions 1 assignee View on GitHub

@rodnymolina is already working on this.

Since Apr 20, 2023.

bug
Dominant language
Shell
Stars
3.9k
Forks
230
Avg merge
7h 48m
Merged PRs (30d)
3

Description

It appears as though during the sysbox install process (via your daemonset) I get some extra cri-o configs 100-crio-bridge.conf, 200-loopback.conf co-located in /etc/cni/net.d that breaks my GKE networking and introduces a subnet for the node that's not correct according to what's been configured in GKE by the 10-gke-ptp.conflist config file already there.

When pods come back online after the kubelet restart on that node, a number of them including the sysbox-installer get this incorrect CIDR provided in 100-crio-bridge.conf and they go into crash loop backoff state as they cannot reach anything on the network (which is correct, there are no firewall/policy rules allowing traffic from this new CIDR that came out of thin air.)

For reference, here's the content of 100-crio-bridge.conf that gets placed into /etc/cni/net.d and breaks the networking for pods running on the node I'm trying to install sysbox on:

{
    "cniVersion": "0.3.1",
    "name": "crio",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "hairpinMode": true,
    "ipam": {
        "type": "host-local",
        "routes": [
            { "dst": "0.0.0.0/0" },
            { "dst": "1100:200::1/24" }
        ],
        "ranges": [
            [{ "subnet": "10.85.0.0/16" }],
            [{ "subnet": "1100:200::/24" }]
        ]
    }
}

I've tried restarting various services on the node, along with just restarting the node entirely after deleting these extraneous files from /etc/cni/net.d (while ensuring I keep 10-gke-ptp.conflist) but no dice, I just get a ton of messages like this:

Apr 19 21:27:34 gha-std-runners-2-baae0455-8zr5 kubelet[25451]: E0419 21:27:34.993101   25451 pod_workers.go:965] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/cni/net.d/. Has your network provider started?" pod="kube-system/sysbox-deploy-k8s-98fpk" podUID=3287c25f-d5f2-477e-a934-918722a0908a

@rodnymolina for viz as you asked me to bring this in as an Issue to track.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.