abiosoft / abiosoft/colima

Istio CNI plugin not working on Colima 0.4.6 with docker runtime

Open
#448 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.8k
Forks
613
PR merge metrics
No merged PRs in 30d

Description

### Description

When using colima to run a Kubernetes cluster with Istio + its CNI plugin installed, pods which inject the Istio sidecar do not work.

I believe this may be related to https://github.com/abiosoft/colima/issues/385 or perhaps it is even the same issue. In any event, maybe this report will serve as a way to reproduce in a concise fashion.

### Version

**Colima Version:**
colima version v0.4.6
git commit: 10377f3a20c2b0f7196ad5944264b69f048a3d40

**Lima Version:**
limactl version 0.11.3

**Qemu Version:**
qemu-img version 7.1.0
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

### Operating System

- [X] macOS Intel
- [ ] macOS M1
- [ ] Linux

### Reproduction Steps

1. Start colima with the docker runtime and kubernetes enabled
```
=> colima start --runtime docker --kubernetes --kubernetes-version "v1.22.15+k3s1" --cpu 4 --memory 16 --disk 40
INFO[0000] starting colima
INFO[0000] runtime: docker+k3s
INFO[0000] preparing network ... context=vm
INFO[0000] creating and starting ... context=vm
INFO[0030] provisioning ... context=docker
INFO[0030] starting ... context=docker
INFO[0036] provisioning ... context=kubernetes
INFO[0036] downloading and installing ... context=kubernetes
INFO[0046] loading oci images ... context=kubernetes
INFO[0054] starting ... context=kubernetes
INFO[0058] updating config ... context=kubernetes
INFO[0059] Switched to context "colima". context=kubernetes
INFO[0059] done
```

2. Install Istio into the k8s cluster, pointing to the appropriate CNI directories used by k3s
```
=> istioctl version
no running Istio pods in "istio-system"
1.13.7

=> istioctl install --set 'components.cni.enabled=true' --set 'values.cni.cniBinDir=/var/lib/rancher/k3s/data/current/bin' --set 'values.cni.cniConfDir=/var/lib/rancher/k3s/agent/etc/cni/net.d'
This will install the Istio 1.13.7 default profile with ["Istio core" "Istiod" "CNI" "Ingress gateways"] components into the cluster. Proceed? (y/N) y
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ CNI installed
✔ Installation complete Making this installation the default for injection and validation.

Thank you for installing Istio 1.13. Please take a few minutes to tell us about your install/upgrade experience! https://forms.gle/pzWZpAvMVBecaQ9h9
```

Note here that the `istio-cni-node` pod seems as if it runs correctly here and the logs don't indicate any problems, however...

3. Label the default namespace and deploy an arbitrary pod for which Istio will inject its proxy sidecar.
```
=> kubectl label namespace default "istio-injection=enabled"
namespace/default labeled

=> kubectl create deployment caddy-app --image caddy
deployment.apps/caddy-app created
```

Due to the use of Istio's CNI plugin, the pod has an `istio-validation` initContainer, and it's this container that encounters the problem, which appears as:
```
=> kubectl logs -lapp=caddy-app -c istio-validation
2022-10-17T20:26:46.727897Z info in new validator: 172.17.0.9
2022-10-17T20:26:46.728051Z info Listening on 127.0.0.1:15001
2022-10-17T20:26:46.728439Z info Listening on 127.0.0.1:15006
2022-10-17T20:26:46.729492Z error Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
2022-10-17T20:26:47.730360Z error Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
2022-10-17T20:26:48.732943Z error Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
2022-10-17T20:26:49.736389Z error Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
2022-10-17T20:26:50.740989Z error Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
2022-10-17T20:26:51.728686Z error validation timeout
```

### Expected behaviour

The `istio-validation` initContainer should succeed and then allow the other containers in the pod to start.

### Additional context

I have also tried this same scenario using other (newer) k8s versions as well as the containerd runtime instead, but I encountered this issue in all cases.

The issue I linked above suggested that this was working on colima 0.3.x - I haven't tried that yet so I'll give it a shot and report back with my findings.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue involves Istio CNI plugin failing on Colima with docker runtime. Start by examining the CNI configuration directories (/var/lib/rancher/k3s/data/current/bin and /var/lib/rancher/k3s/agent/etc/cni/net.d) inside the Colima VM. Check the network setup and compare with known working versions (Colima 0.3.x). Look at the istio-validation initContainer logs to understand the connection refusal to 127.0.0.6:15002. Verify CNI plugin installation and network namespace handling in the Colima environment. Done looks like pods with Istio sidecar injection start successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.