nftables: support an nftables-based egress-firewall sidecar under runsc (OpenSandbox) — use case + tracking PRs
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
## Why we need this (use case)
[OpenSandbox](https://github.com/opensandbox-group/OpenSandbox) runs **untrusted, AI-generated code** in ephemeral, per-request pods on Kubernetes. gVisor (GKE Sandbox) is the isolation runtime we want for exactly this "safely run untrusted code" scenario.
Each sandbox pod ships an **egress-firewall sidecar** (runs with `CAP_NET_ADMIN`) that enforces an outbound network policy so the untrusted workload can only reach an allowlist. It programs the dataplane entirely with **nftables**:
- a `nat` / `output` **DNS redirect** to a local DNS proxy (`redirect to :15353`), with a `meta mark` bypass for the proxy's own upstream queries;
- a transparent-HTTP **redirect** for MITM inspection (`meta skuid != `, `tcp dport {80,443}`, `redirect to :`);
- an `inet` filter chain on the `output` hook implementing a domain→IP allowlist: **interval sets** (CIDR allow/deny), **timeout sets** (dynamically-resolved IPs that expire), `ct state established,related accept`, `meta mark accept`, `oifname "lo" accept`, and `ip/ip6 daddr @set` lookups.
Under runsc this ruleset could not be installed because several nft features were unimplemented, so the egress sidecar could not run under gVisor at all.
### It works under Kata, but we want gVisor
This exact setup **works today under Kata Containers** — Kata gives each pod a full Linux kernel, so the egress sidecar's `nat`/`REDIRECT` rule works — but Kata puts a full micro-VM under every sandbox, i.e. much higher cold-start latency and per-pod memory/density overhead, which is expensive for ephemeral, per-request sandboxes. We want gVisor's lighter-weight, syscall-level isolation instead.
OpenSandbox documents this gVisor-vs-Kata situation:
- Egress sidecar incompatible with gVisor, with Kata as the documented workaround: [`docs/guides/secure-container.md` §"Egress Sidecar Incompatible with gVisor"](https://github.com/opensandbox-group/OpenSandbox/blob/main/docs/guides/secure-container.md#5-egress-sidecar-incompatible-with-gvisor) (same doc also has the runtime cold-start / overhead comparison table)
- Network-isolation rationale — the egress sidecar's `nat`-table `REDIRECT` works under `runc` and all Kata variants (`kata-qemu`/`kata-clh`/`kata-fc`) but not gVisor: [`docs/architecture/network-isolation.md` §"Runtime Compatibility"](https://github.com/opensandbox-group/OpenSandbox/blob/main/docs/architecture/network-isolation.md#runtime-compatibility)
- Kata deployment example: [`docs/examples/aks-kata.md`](https://github.com/opensandbox-group/OpenSandbox/blob/main/docs/examples/aks-kata.md#aks-kata-example)
(Those docs attribute the gVisor limitation to the iptables `nat` table / gvisor#170. The PRs below provide the **native-nftables** path — `redir` plus the set backends — so the egress can run under gVisor without relying on the iptables `nat` table at all.)
## What was missing, and the PRs
Running the egress's exact ruleset under `runsc --TESTONLY-nftables` (Docker + `nft`), the gaps were as follows (some pieces — `ct state`, `oifname`, `meta skuid`, set lookups, verdicts — already worked):
| Gap | PR |
|---|---|
| `meta mark` load | #13797 |
| interval sets (CIDR allow/deny) | #13799 |
| timeout sets (expiring dynamic allow) | #13800 |
| `redir` / `REDIRECT` (DNS + MITM redirect) | #13801 |
Each PR includes unit tests plus end-to-end validation under runsc. The `redir` change is additionally validated **functionally**: with a redirect rule installed, a datagram sent to `8.8.8.8:53` is delivered to a local listener bound to `127.0.0.1:15353`.
With these four, the entire nft policy engine the egress needs works under gVisor — **no `iptables-nft`/xt-compat is required**. The matching OpenSandbox change that makes the egress emit native nft for the DNS and MITM redirects (instead of shelling out to iptables) is **[opensandbox-group/OpenSandbox#1374](https://github.com/opensandbox-group/OpenSandbox/pull/1374)** — kept in draft until these gVisor PRs land and a runsc release ships with them.
## Suggested merge order
- **#13797** (`meta mark`) — independent (touches `nft_meta.go` / `nft_metaload.go`); can merge any time.
- **#13799** (interval sets) → **#13800** (timeout sets) → **#13801** (`redir`) — these touch the shared `pkg/tcpip/nftables` package and are stacked in that order. Please merge in sequence; each is rebased onto `master` after the previous lands.
## Notes / out of scope
- Set-element deletion (`NFT_MSG_DELSETELEM`, #13789) is **not** required by this use case — the egress relies on set timeouts for expiry, not explicit deletes — and is orthogonal.
- Relates to the umbrella #11778 (NETLINK_NETFILTER support) and #10510.
Contributor guide
Research direction
Start with the referenced PRs #13797, #13799, #13800, and #13801, then inspect nft_meta.go, nft_metaload.go, and the shared pkg/tcpip/nftables package. Run the nftables tests and the documented runsc --TESTONLY-nftables validation; the tracking work is done when the four capabilities and the OpenSandbox native-nft path work together under runsc.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, linux
- Domain
- networking, operating-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100