containerd / containerd/nerdctl

[Lima? CNI? (w/ rootless?)] network degrading over time?

Open
#3,487 3 comments 0 reactions 0 assignees View on GitHub
kind/external/lima
Dominant language
Go
Stars
10.4k
Forks
826
Avg merge
1d 23h
Merged PRs (30d)
44

Description

Is there a network guru here who could advise on how to further debug this?

### Description

After heavy, prolonged usage and testing of nerdctl, network inside lima seems to be degrading, with a very large proportion of all requests ending with i/o timeout.

This is affecting the entire VM networking, not just nerdctl. Rebooting the VM does not help.

The same requests ran from the host (or from another VM) are just fine.

It is unclear to me if this would be a lima issue, a cni issue, or a nerdctl issue?

Something as simple as
`curl https://ghcr.io/v2/stargz-containers/registry/manifests/2-org`

Will intermittently (~70% of the time) fail with:

```
curl: (28) Failed to connect to ghcr.io port 443 after 132561 ms: Couldn't connect to server
```

tcpdump:
```
11:56:15.724007 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029123835 ecr 0,nop,wscale 7], length 0
11:56:16.769207 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029124880 ecr 0,nop,wscale 7], length 0
11:56:17.794140 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029125905 ecr 0,nop,wscale 7], length 0
11:56:18.812797 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029126924 ecr 0,nop,wscale 7], length 0
11:56:19.842554 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029127953 ecr 0,nop,wscale 7], length 0
11:56:20.860340 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029128971 ecr 0,nop,wscale 7], length 0
11:56:22.906894 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029131018 ecr 0,nop,wscale 7], length 0
11:56:26.942212 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029135053 ecr 0,nop,wscale 7], length 0
11:56:35.133635 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029143245 ecr 0,nop,wscale 7], length 0
11:56:51.515252 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029159626 ecr 0,nop,wscale 7], length 0
11:57:23.775608 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029191886 ecr 0,nop,wscale 7], length 0
```

iptables-save
```
# Generated by iptables-save v1.8.10 (nf_tables) on Wed Oct 2 12:00:27 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:CNI-ADMIN - [0:0]
:CNI-FORWARD - [0:0]
:CNI-ISOLATION-STAGE-1 - [0:0]
:CNI-ISOLATION-STAGE-2 - [0:0]
-A FORWARD -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j CNI-ISOLATION-STAGE-1
-A FORWARD -m comment --comment "CNI firewall plugin rules" -j CNI-FORWARD
-A CNI-FORWARD -m comment --comment "CNI firewall plugin admin overrides" -j CNI-ADMIN
-A CNI-ISOLATION-STAGE-1 -i nerdctl0 ! -o nerdctl0 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j CNI-ISOLATION-STAGE-2
-A CNI-ISOLATION-STAGE-1 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j RETURN
-A CNI-ISOLATION-STAGE-2 -o nerdctl0 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j DROP
-A CNI-ISOLATION-STAGE-2 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j RETURN
COMMIT
# Completed on Wed Oct 2 12:00:27 2024
# Generated by iptables-save v1.8.10 (nf_tables) on Wed Oct 2 12:00:27 2024
*nat
:PREROUTING ACCEPT [4:1843]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [72782:32720052]
:POSTROUTING ACCEPT [72782:32720052]
:CNI-53bc5ebfdf1a5ca6fc355b8a - [0:0]
:CNI-bca742bf74f55524d8dda11b - [0:0]
:LIMADNS - [0:0]
-A PREROUTING -j LIMADNS
-A OUTPUT -j LIMADNS
-A POSTROUTING -s 10.4.0.21/32 -m comment --comment "name: \"bridge\" id: \"default-e1839c3f7e677f2e525c4476b5d504ebd9f8368387a5dbf3bfd69c4b187e9147\"" -j CNI-bca742bf74f55524d8dda11b
-A POSTROUTING -s 10.4.0.22/32 -m comment --comment "name: \"bridge\" id: \"default-516fc37228b72aee23de771c219bedd9510ca3af0f5d0d6ec42847f180848422\"" -j CNI-53bc5ebfdf1a5ca6fc355b8a
-A CNI-53bc5ebfdf1a5ca6fc355b8a -d 10.4.0.0/24 -m comment --comment "name: \"bridge\" id: \"default-516fc37228b72aee23de771c219bedd9510ca3af0f5d0d6ec42847f180848422\"" -j ACCEPT
-A CNI-53bc5ebfdf1a5ca6fc355b8a ! -d 224.0.0.0/4 -m comment --comment "name: \"bridge\" id: \"default-516fc37228b72aee23de771c219bedd9510ca3af0f5d0d6ec42847f180848422\"" -j MASQUERADE
-A CNI-bca742bf74f55524d8dda11b -d 10.4.0.0/24 -m comment --comment "name: \"bridge\" id: \"default-e1839c3f7e677f2e525c4476b5d504ebd9f8368387a5dbf3bfd69c4b187e9147\"" -j ACCEPT
-A CNI-bca742bf74f55524d8dda11b ! -d 224.0.0.0/4 -m comment --comment "name: \"bridge\" id: \"default-e1839c3f7e677f2e525c4476b5d504ebd9f8368387a5dbf3bfd69c4b187e9147\"" -j MASQUERADE
COMMIT
# Completed on Wed Oct 2 12:00:27 2024
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.