apple / apple/containerization

[Request]: Enable Cilium's base kernel requirements in the kernel configs

Open
#912 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
8.9k
Forks
359
Avg merge
2d 20h
Merged PRs (30d)
13

Description

### Feature or enhancement request details

`kernel/config-arm64` (and `config-x86_64`) is the documented way to build a custom kernel for `container`, and apple/container discussion #1673 asks for Cilium in `container k8s` clusters and links Cilium's requirements page. Cilium's documented base requirements (https://docs.cilium.io/en/stable/operations/system_requirements/#base-requirements):

```
CONFIG_BPF=y CONFIG_BPF_EVENTS=y CONFIG_BPF_SYSCALL=y CONFIG_NET_CLS_BPF=y CONFIG_BPF_JIT=y
CONFIG_NET_CLS_ACT=y CONFIG_NET_SCH_INGRESS=y CONFIG_DEBUG_INFO_BTF=y CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_USER_API_HASH=y CONFIG_CGROUPS=y CONFIG_CGROUP_BPF=y CONFIG_PERF_EVENTS=y CONFIG_SCHEDSTATS=y
```

Five are not enabled in `config-arm64` on `main` (the same five in `config-x86_64`):

| Option | State today | Note |
|---|---|---|
| `CONFIG_BPF_JIT` | absent | the Cilium agent needs a JIT |
| `CONFIG_SCHEDSTATS` | `# ... is not set` | one-line flip |
| `CONFIG_CRYPTO_USER_API_HASH` | `# ... is not set` | one-line flip |
| `CONFIG_BPF_EVENTS` | absent | needs `FTRACE` and `KPROBES`/`KPROBE_EVENTS`, otherwise olddefconfig drops it |
| `CONFIG_DEBUG_INFO_BTF` | absent (`DEBUG_INFO_NONE=y`) | needs `pahole` (dwarves) in `kernel/image` |

Everything else on the list, including the tunneling, netfilter, L7 and WireGuard sets, is already `=y`. `CONFIG_NETKIT=y` (Cilium's netkit device mode, optional) is also absent. For comparison, the kernel `container` recommends today (Kata 3.32.0 debug, 6.18.35) has `BPF_JIT`, `BPF_EVENTS` and `DEBUG_INFO_BTF` but not `SCHEDSTATS`, `NETKIT` or `NETFILTER_XT_MATCH_SOCKET`. A Cilium 1.19.4 agent starts on it but keeps failing to install its proxy iptables rules (`-m socket --transparent`). This request is only about `kernel/config-arm64`.

**Measured.** I built linux-6.18.5 (the version `kernel/Makefile` pins) three times from `config-arm64`, same toolchain for all three (gcc 13.3 in ubuntu:24.04, not the gcc 9.4 `kernel/image` toolchain the file was generated with, so absolute sizes will differ slightly from an in-tree build; the deltas should not):

| Variant | arm64 `Image` bytes | Delta |
|---|---|---|
| `config-arm64` as-is | 29,354,496 | |
| + `BPF_JIT`, `SCHEDSTATS`, `CRYPTO_USER_API_HASH`, `NETKIT` | 29,424,128 | +69,632 (+0.24%) |
| + the above and `BPF_EVENTS` with `FTRACE`, `KPROBES`, `KPROBE_EVENTS` | 33,769,984 | +4,415,488 (+15.0%) |

`make olddefconfig` on the hand-edited small variant adds `BPF_JIT_DEFAULT_ON=y` and `EXECMEM=y` and surfaces `# CONFIG_BPF_JIT_ALWAYS_ON is not set`; nothing else moves relative to the regenerated baseline. The tracing variant additionally pulls in `TRACING`, `EVENT_TRACING`, `TRACEPOINTS`, `RING_BUFFER`, `KRETPROBES`, `UPROBES`/`UPROBE_EVENTS`, `EPROBE_EVENTS`, `DYNAMIC_EVENTS`, `NOP_TRACER`, `CONTEXT_SWITCH_TRACER`, `GLOB`, `TRACE_CLOCK` (all tracers stay off).

Functional check, both variants, on container CLI 1.0.0: booted with `container run --kernel` (`bpf_jit_enable=1`, `/proc/schedstat` present, an `AF_ALG` hash socket binds, `ip link add ... type netkit` succeeds where 6.12.28 and 6.18.35 both say "Unknown device type"), then a two-node kubeadm 1.37 cluster on kindest/node with Cilium 1.19.4 via `cilium install --wait`: `cilium status` OK with 2/2 agents, nodes Ready, and from a curl pod ClusterIP, pod IP and cluster DNS answer 200, including pod-to-pod across nodes with `cilium-dbg status` reporting `Routing: Tunnel [vxlan]`. The small variant is enough; the tracing variant behaved the same, and its tracing features themselves were not exercised at runtime (size is the only thing measured for it).

**Proposal**

1. Enable `CONFIG_BPF_JIT`, `CONFIG_SCHEDSTATS`, `CONFIG_CRYPTO_USER_API_HASH` and `CONFIG_NETKIT`, hand-edited in place the way #681 and #797 were: +68 KB, and enough for the Cilium agent. PR attached, both arches (only arm64 built and tested).
2. Decide separately on the two heavier items Cilium also lists: `CONFIG_BPF_EVENTS` (+4.4 MB with its tracing dependencies; needed by kprobe- and tracepoint-based BPF programs such as Tetragon) and `CONFIG_DEBUG_INFO_BTF` (needs `pahole` in `kernel/image` and flips `DEBUG_INFO_NONE`; needed by CO-RE tooling like `bpftool` and libbpf programs). Neither was needed for Cilium 1.19.4 here, so they fit a follow-up with measured boot-time cost, or a documented custom-build fragment.

### Code of Conduct
- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start with kernel/config-arm64 and compare kernel/config-x86_64, using the linux-6.18.5 version pinned in kernel/Makefile. Run make olddefconfig after the configuration changes, then use container run --kernel and the described Cilium/kubeadm checks; done means the four proposed options are enabled and the documented Cilium checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, linux
Domain
networking, operating-systems
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.