google / google/gvisor

nftables: support an nftables-based egress-firewall sidecar under runsc (OpenSandbox) — use case + tracking PRs

Open
#13,796 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

## Why we need this (use case)

[OpenSandbox](https://github.com/opensandbox-group/OpenSandbox) runs **untrusted, AI-generated code** in ephemeral, per-request pods on Kubernetes. gVisor (GKE Sandbox) is the isolation runtime we want for exactly this "safely run untrusted code" scenario.

Each sandbox pod ships an **egress-firewall sidecar** (runs with `CAP_NET_ADMIN`) that enforces an outbound network policy so the untrusted workload can only reach an allowlist. It programs the dataplane entirely with **nftables**:

- a `nat` / `output` **DNS redirect** to a local DNS proxy (`redirect to :15353`), with a `meta mark` bypass for the proxy's own upstream queries;
- a transparent-HTTP **redirect** for MITM inspection (`meta skuid != `, `tcp dport {80,443}`, `redirect to :`);
- an `inet` filter chain on the `output` hook implementing a domain→IP allowlist: **interval sets** (CIDR allow/deny), **timeout sets** (dynamically-resolved IPs that expire), `ct state established,related accept`, `meta mark accept`, `oifname "lo" accept`, and `ip/ip6 daddr @set` lookups.

Under runsc this ruleset could not be installed because several nft features were unimplemented, so the egress sidecar could not run under gVisor at all.

### It works under Kata, but we want gVisor

This exact setup **works today under Kata Containers** — Kata gives each pod a full Linux kernel, so the egress sidecar's `nat`/`REDIRECT` rule works — but Kata puts a full micro-VM under every sandbox, i.e. much higher cold-start latency and per-pod memory/density overhead, which is expensive for ephemeral, per-request sandboxes. We want gVisor's lighter-weight, syscall-level isolation instead.

OpenSandbox documents this gVisor-vs-Kata situation:

- Egress sidecar incompatible with gVisor, with Kata as the documented workaround: [`docs/guides/secure-container.md` §"Egress Sidecar Incompatible with gVisor"](https://github.com/opensandbox-group/OpenSandbox/blob/main/docs/guides/secure-container.md#5-egress-sidecar-incompatible-with-gvisor) (same doc also has the runtime cold-start / overhead comparison table)
- Network-isolation rationale — the egress sidecar's `nat`-table `REDIRECT` works under `runc` and all Kata variants (`kata-qemu`/`kata-clh`/`kata-fc`) but not gVisor: [`docs/architecture/network-isolation.md` §"Runtime Compatibility"](https://github.com/opensandbox-group/OpenSandbox/blob/main/docs/architecture/network-isolation.md#runtime-compatibility)
- Kata deployment example: [`docs/examples/aks-kata.md`](https://github.com/opensandbox-group/OpenSandbox/blob/main/docs/examples/aks-kata.md#aks-kata-example)

(Those docs attribute the gVisor limitation to the iptables `nat` table / gvisor#170. The PRs below provide the **native-nftables** path — `redir` plus the set backends — so the egress can run under gVisor without relying on the iptables `nat` table at all.)

## What was missing, and the PRs

Running the egress's exact ruleset under `runsc --TESTONLY-nftables` (Docker + `nft`), the gaps were as follows (some pieces — `ct state`, `oifname`, `meta skuid`, set lookups, verdicts — already worked):

| Gap | PR |
|---|---|
| `meta mark` load | #13797 |
| interval sets (CIDR allow/deny) | #13799 |
| timeout sets (expiring dynamic allow) | #13800 |
| `redir` / `REDIRECT` (DNS + MITM redirect) | #13801 |

Each PR includes unit tests plus end-to-end validation under runsc. The `redir` change is additionally validated **functionally**: with a redirect rule installed, a datagram sent to `8.8.8.8:53` is delivered to a local listener bound to `127.0.0.1:15353`.

With these four, the entire nft policy engine the egress needs works under gVisor — **no `iptables-nft`/xt-compat is required**. The matching OpenSandbox change that makes the egress emit native nft for the DNS and MITM redirects (instead of shelling out to iptables) is **[opensandbox-group/OpenSandbox#1374](https://github.com/opensandbox-group/OpenSandbox/pull/1374)** — kept in draft until these gVisor PRs land and a runsc release ships with them.

## Suggested merge order

- **#13797** (`meta mark`) — independent (touches `nft_meta.go` / `nft_metaload.go`); can merge any time.
- **#13799** (interval sets) → **#13800** (timeout sets) → **#13801** (`redir`) — these touch the shared `pkg/tcpip/nftables` package and are stacked in that order. Please merge in sequence; each is rebased onto `master` after the previous lands.

## Notes / out of scope

- Set-element deletion (`NFT_MSG_DELSETELEM`, #13789) is **not** required by this use case — the egress relies on set timeouts for expiry, not explicit deletes — and is orthogonal.
- Relates to the umbrella #11778 (NETLINK_NETFILTER support) and #10510.

Contributor guide

Open the contributing guide

Research direction

Start with the referenced PRs #13797, #13799, #13800, and #13801, then inspect nft_meta.go, nft_metaload.go, and the shared pkg/tcpip/nftables package. Run the nftables tests and the documented runsc --TESTONLY-nftables validation; the tracking work is done when the four capabilities and the OpenSandbox native-nft path work together under runsc.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, linux
Domain
networking, operating-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.