google / google/gvisor

Run stock k3s in gVisor

Open
#14,637 5 comments 3 reactions 1 assignee Claimed by @hugelgupf View on GitHub
type: enhancement
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

### Description

Hey folks,

so I'm doing something slightly crazy, but I have some test infra that runs in k3s. That infra I want to be running in gVisor.

I've got a stock k3s+kubeproxy+bridge+flannel working in a local setup.

I've sent a few patches already to make this work:

## Faithful implementations of Linux

| Upstream | What it is | Why needed |
| --- | --- | --- |
| ✅ [google/gvisor#14600](https://github.com/google/gvisor/pull/14600) (open) | Raise `setsockopt` `maxOptLen` 32KB→1MB | `IPT_SO_SET_REPLACE` passes the whole nat table in one call; kube-proxy's full sut-setup table exceeds 32KB and was rejected at the syscall layer |
| ✅ [#14601](https://github.com/google/gvisor/pull/14601) (open) | Report `IFLA_OPERSTATE` in `RTM_GETLINK` | bridge CNI polls link operstate to confirm a veth is up |
| ✅ [#14602](https://github.com/google/gvisor/pull/14602) (open) | iptables `comment` match extension | nearly every kube-proxy rule carries `-m comment`; without it the whole restore fails |
| ✅ [#14603](https://github.com/google/gvisor/pull/14603) (open) | iptables `conntrack` match (revs 1/2/3) | kube-proxy uses `-m conntrack --ctstate` and negotiates rev 3 |
| ✅ [#14604](https://github.com/google/gvisor/pull/14604) (open) | iptables `addrtype` match | kube-proxy's KUBE-SERVICES rules use `-m addrtype --dst-type LOCAL` |
| ✅ [#14605](https://github.com/google/gvisor/pull/14605) (open) | `MARK` target + `RuleContinue` verdict + full `mangle` table hooks | kube-proxy marks masquerade-bound packets (`--set-xmark 0x4000`) in mangle and matches later |
| ✅ [#14606](https://github.com/google/gvisor/pull/14606) (open) | `MASQUERADE` target (layout-agnostic: 56-byte compat + 72-byte `nf_nat_range`) | kube-proxy's KUBE-POSTROUTING SNAT + `--random-fully` |
| ✅ [#14624](https://github.com/google/gvisor/pull/14624) (open) | Make `net.ipv4.ip_forward` writable (0644), backed by `stack.SetForwarding` | kubelet/kube-proxy enable IPv4 forwarding at startup |
| ✅ [#14625](https://github.com/google/gvisor/pull/14625) (open) | Writable `net.ipv4.conf.*.route_localnet` (AllowExternalLoopbackTraffic) | kube-proxy sets `route_localnet=1` for NodePort/localhost service handling |
| ✅ [#14626](https://github.com/google/gvisor/pull/14626) (open) | Route post-OUTPUT-DNAT packets by destination (empty `localAddr` to `FindRoute`) | node→ClusterIP: pre-DNAT source picked the wrong NIC on a multi-NIC node, dropping the SYN |

## Stubs

There's a few things that I have working locally, but that aren't faithful implementations of Linux yet that I'd like opinions on:

* bridge-netfilter -- I have a partial implementation of bridge-nf-call-iptables sysctl; only handles the reverse-NAT path right now. If nobody else has something in the works, I may try to expand that to a fully faithful implementation.

* related: bridge CNI sets hairpin mode using RTM_SETLINK / IFLA_PROTINFO, I'm ignoring that right now

* kubelet depends on writing to some kernel [tunables](https://github.com/kubernetes/kubernetes/blob/5261422c0c345347c1469d16148fd8539d7e60f4/pkg/kubelet/cm/container_manager_linux.go#L477) that I've just stubbed out for now because they don't make sense in gVisor (e.g. /proc/sys/panic)

### Is this feature related to a specific bug?

_No response_

### Do you have a specific solution in mind?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.