NVIDIA / NVIDIA/OpenShell

feat(supervisor): NFLOG-based bypass detection to drop the CAP_SYSLOG requirement

Open
#2,382 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:sandbox area:supervisor state:stale topic:security
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

Problem Statement

Sandbox pods/containers currently request CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_SYS_PTRACE, and CAP_SYSLOG. With Kubernetes user namespaces (GA in 1.36, already supported via enable_user_namespaces) the first three become namespaced and lose their host-level power — but CAP_SYSLOG does not. The kernel checks syslog(2)//dev/kmsg access against the initial user namespace (capable(), not ns_capable()), so in a hostUsers: false pod the granted CAP_SYSLOG is a no-op and the bypass detection monitor silently degrades.

The only consumer of CAP_SYSLOG is the bypass detection monitor in openshell-supervisor-process, which tails dmesg --follow for nftables log entries (prefix openshell:bypass:<ns>:) emitted by the per-workload netns REJECT rules. Reading the host-global kernel ring buffer from inside a sandbox is also a mild information leak, and requires the nf_log_all_netns sysctl on the node.

Proposed Design

Switch bypass log delivery from kernel-printk to nfnetlink_log (NFLOG):

  • During netns setup (privileged startup, before supervisor seccomp hardening), bind an AF_NETLINK/NETLINK_NETFILTER socket to an NFLOG group inside the workload netns. Netns affinity is fixed at socket() time; binding needs only CAP_NET_ADMIN over the netns owner, which works under user namespaces.
  • Generate the nftables log rules as log prefix <p> group <N> when the socket bind succeeds; keep the existing log ... flags skuid printk rules as an automatic fallback when it fails (e.g. kernel without nfnetlink_log).
  • The bypass monitor consumes NFLOG packet messages (prefix attribute for filtering, NFULA_UID for workload UID, raw IP payload for dst/ports/proto) and reports through the existing OCSF dual-emit path. The dmesg path remains as fallback (still useful in the VM driver, which runs with full capabilities).
  • Drop CAP_SYSLOG from the Kubernetes, Docker, and Podman driver capability sets.

Enforcement is unaffected: the nftables REJECT rules do not depend on the log delivery mechanism. NFLOG delivery is netns-scoped, so the monitor no longer reads host-global kernel logs, and nf_log_all_netns is no longer needed on the NFLOG path.

Alternatives Considered

  • Keep dmesg, relax kernel.dmesg_restrict on nodes — exposes host kernel logs to every pod on the node; strictly worse.
  • eBPF-based detectionbpf() is also initial-userns-gated (CAP_BPF/CAP_SYS_ADMIN in init ns), so it has the same problem as CAP_SYSLOG, plus a new dependency.
  • Netlink crate dependency (netlink-packet-netfilter / libnetfilter_log bindings) — the required protocol subset (group bind, copy mode, packet attr parsing) is ~stable kernel ABI and small; a hand-rolled libc implementation avoids new dependencies in a security-sensitive crate and matches its existing raw-FFI style.

Agent Investigation

  • CAP_SYSLOG's only consumer is bypass_monitor::spawn (crates/openshell-supervisor-process/src/bypass_monitor/mod.rs); drivers request it in openshell-driver-kubernetes/src/driver.rs, openshell-driver-docker/src/lib.rs, openshell-driver-podman/src/container.rs.
  • No eBPF usage anywhere in the workspace; netns/nftables management shells out to iproute2/nft, so NFLOG covers the whole detection path.
  • The supervisor seccomp prelude does not restrict socket(); the workload runtime filter blocks non-NETLINK_ROUTE netlink sockets, so the sandboxed workload cannot bind the NFLOG group itself.
  • The sidecar topology has no bypass log listener today (no CAP_SYSLOG there either); it keeps printk rules unchanged.
  • Implementation is complete on a branch with unit tests (netlink message layout, datagram parsing, IPv4/IPv6 packet parsing, ruleset generation); Linux cargo check/clippy (pedantic+nursery)/tests pass. PR to follow.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/openshell-supervisor-process/src/bypass_monitor/mod.rs and the driver capability files in openshell-driver-kubernetes/src/driver.rs, openshell-driver-docker/src/lib.rs, and openshell-driver-podman/src/container.rs. Review the existing branch implementation and its unit tests, then run Linux cargo check, clippy, and tests. Done means NFLOG delivery works with printk fallback and CAP_SYSLOG is removed from the listed drivers.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, linux, rust
Domain
networking, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.