Run stock k3s in gVisor
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
### Description
Hey folks,
so I'm doing something slightly crazy, but I have some test infra that runs in k3s. That infra I want to be running in gVisor.
I've got a stock k3s+kubeproxy+bridge+flannel working in a local setup.
I've sent a few patches already to make this work:
## Faithful implementations of Linux
| Upstream | What it is | Why needed |
| --- | --- | --- |
| ✅ [google/gvisor#14600](https://github.com/google/gvisor/pull/14600) (open) | Raise `setsockopt` `maxOptLen` 32KB→1MB | `IPT_SO_SET_REPLACE` passes the whole nat table in one call; kube-proxy's full sut-setup table exceeds 32KB and was rejected at the syscall layer |
| ✅ [#14601](https://github.com/google/gvisor/pull/14601) (open) | Report `IFLA_OPERSTATE` in `RTM_GETLINK` | bridge CNI polls link operstate to confirm a veth is up |
| ✅ [#14602](https://github.com/google/gvisor/pull/14602) (open) | iptables `comment` match extension | nearly every kube-proxy rule carries `-m comment`; without it the whole restore fails |
| ✅ [#14603](https://github.com/google/gvisor/pull/14603) (open) | iptables `conntrack` match (revs 1/2/3) | kube-proxy uses `-m conntrack --ctstate` and negotiates rev 3 |
| ✅ [#14604](https://github.com/google/gvisor/pull/14604) (open) | iptables `addrtype` match | kube-proxy's KUBE-SERVICES rules use `-m addrtype --dst-type LOCAL` |
| ✅ [#14605](https://github.com/google/gvisor/pull/14605) (open) | `MARK` target + `RuleContinue` verdict + full `mangle` table hooks | kube-proxy marks masquerade-bound packets (`--set-xmark 0x4000`) in mangle and matches later |
| ✅ [#14606](https://github.com/google/gvisor/pull/14606) (open) | `MASQUERADE` target (layout-agnostic: 56-byte compat + 72-byte `nf_nat_range`) | kube-proxy's KUBE-POSTROUTING SNAT + `--random-fully` |
| ✅ [#14624](https://github.com/google/gvisor/pull/14624) (open) | Make `net.ipv4.ip_forward` writable (0644), backed by `stack.SetForwarding` | kubelet/kube-proxy enable IPv4 forwarding at startup |
| ✅ [#14625](https://github.com/google/gvisor/pull/14625) (open) | Writable `net.ipv4.conf.*.route_localnet` (AllowExternalLoopbackTraffic) | kube-proxy sets `route_localnet=1` for NodePort/localhost service handling |
| ✅ [#14626](https://github.com/google/gvisor/pull/14626) (open) | Route post-OUTPUT-DNAT packets by destination (empty `localAddr` to `FindRoute`) | node→ClusterIP: pre-DNAT source picked the wrong NIC on a multi-NIC node, dropping the SYN |
## Stubs
There's a few things that I have working locally, but that aren't faithful implementations of Linux yet that I'd like opinions on:
* bridge-netfilter -- I have a partial implementation of bridge-nf-call-iptables sysctl; only handles the reverse-NAT path right now. If nobody else has something in the works, I may try to expand that to a fully faithful implementation.
* related: bridge CNI sets hairpin mode using RTM_SETLINK / IFLA_PROTINFO, I'm ignoring that right now
* kubelet depends on writing to some kernel [tunables](https://github.com/kubernetes/kubernetes/blob/5261422c0c345347c1469d16148fd8539d7e60f4/pkg/kubelet/cm/container_manager_linux.go#L477) that I've just stubbed out for now because they don't make sense in gVisor (e.g. /proc/sys/panic)
### Is this feature related to a specific bug?
_No response_
### Do you have a specific solution in mind?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.