hashgraph / hashgraph/solo-weaver

Cilium loadBalancer.acceleration: best-effort attaches XDP to the host public NIC, flapping carrier on ixgbe (X550) and dropping the node's public IP

Open
#669 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3
Forks
0
Avg merge
3d 2h
Merged PRs (30d)
46

Description

## Summary

`kube cluster install` installs Cilium with `loadBalancer.acceleration: "best-effort"` and `routingMode: native` **without an explicit `devices:` list**. On hardware whose primary/public NIC is an Intel X550 (`ixgbe`), Cilium auto-detects that public, default-route NIC as a datapath device and attaches an **XDP** program to it. The `ixgbe` driver performs a PHY/link reset whenever an XDP program is attached or the agent reconciles, which **flaps the NIC's carrier continuously** (Gained → Lost every ~10s). Because the link never holds carrier long enough for `systemd-networkd` to finish configuring it, the node's **static public IP is never installed and the host loses public connectivity**.

There is currently **no supported way to override the Cilium values** (acceleration, devices, etc.) through `kube cluster install`, because the values file is embedded in the binary and rewritten on every run (details below).

## Environment

- solo-weaver: `kube cluster install` flow (`internal/workflows/steps/step_cilium.go`)
- Cilium: **1.18.1** (`installCiliumCNI("1.18.1")`), installed via `cilium install --wait --version "1.18.1" --values /etc/weaver/cilium-config.yaml`
- Node type: `block` (single-node kubeadm cluster), but this affects any node-type on the same hardware
- NIC: Intel Ethernet Controller **X550**, driver **`ixgbe`**, on the host's public/default-route interface (`eno1`)
- OS: Ubuntu, `systemd-networkd` renderer, netplan with a static public `/31` on the public NIC

## Root cause

The embedded values template (`internal/templates/files/cilium/cilium-config.yaml`) sets:

```yaml
# Routing Configuration
routingMode: native
autoDirectNodeRoutes: true
# (no `devices:` key — Cilium auto-detects datapath devices)

# Load Balancer Configuration
loadBalancer:
mode: dsr
dsrDispatch: opt
algorithm: maglev
acceleration: "best-effort" # <-- attaches XDP to native devices where the driver supports it
```

With `acceleration: best-effort` and no `devices:` scoping, Cilium attaches an XDP program to every auto-detected native device — including the host's public, default-route NIC. `ixgbe` resets the link on XDP attach/reconfigure, so the public NIC's carrier flaps indefinitely.

## Evidence

`systemd-networkd` journal — the public NIC (`eno1`) held carrier continuously for days, then lost it the **instant Cilium's datapath came up**, and has flapped ever since:

```
23:46:33 cilium_net: Gained carrier
23:46:33 cilium_host: Gained carrier
23:46:35 lxc_health: Gained carrier
23:46:36 eno1: Lost carrier <-- public NIC drops the moment Cilium attaches
23:46:36 lxc2eae8d518f40: Gained carrier
23:46:45 eno1: Gained carrier
23:46:46 eno1: Lost carrier
23:46:55 eno1: Gained carrier
23:46:57 eno1: Lost carrier
...repeats Gained→Lost every ~10s indefinitely
```

`networkctl status eno1`:
```
State: no-carrier (configuring)
Online state: offline
Driver: ixgbe
Model: Ethernet Controller X550
```

The cluster NIC (`eno2`, also X550/ixgbe) is **not** auto-detected as the primary datapath device, keeps carrier (`Speed: 10Gbps`, `online`), and is unaffected. `systemctl restart systemd-networkd` and `netplan apply` do not help, because the flap is a driver-level link reset driven by Cilium, not a netplan/networkd configuration problem.

## Why operators can't work around it via configuration

`pkg/software/cilium_installer.go` renders the embedded template to `/etc/weaver/cilium-config.yaml` and **overwrites it unconditionally on every `Configure()`**:

```go
// createCiliumConfigFile() — pkg/software/cilium_installer.go
rendered, _ := templates.Render(ciliumTemplateFile, tmplData) // files/cilium/cilium-config.yaml (go:embed)
...
err = ci.fileManager.WriteFile(configFilePath, []byte(rendered)) // clobbers any operator edits
```

So pre-seeding or hand-editing the values file does not survive a run, and `kube cluster install` exposes no `--cilium-values` / acceleration / devices flag. The only operator workaround today is to reconfigure Cilium **after** `kube cluster install` (e.g. `cilium upgrade --reuse-values --set loadBalancer.acceleration=disabled`), which is fragile and defeats the point of a one-shot provisioner.

## Requested changes (any one would resolve this)

1. **Default `loadBalancer.acceleration: disabled`** (use tc-eBPF instead of XDP). tc attach does not reset the `ixgbe` PHY, so the public NIC stays up. This is the lowest-risk fix and works across mixed hardware.
2. **Scope the datapath to the cluster NIC**: set an explicit `devices:` list (or expose it as a knob) so Cilium never attaches XDP/tc to the host's public/management NIC. On these nodes the cluster/node traffic rides a dedicated private interface, so the public NIC should be excluded from the datapath entirely.
3. **Expose Cilium values overrides** on `kube cluster install` (e.g. a `--cilium-values ` passthrough or a config block), so operators can set `acceleration`/`devices` without forking the binary. Today the embedded file is overwritten every run, so there is no seam.

Preferably (1) as the safe default, plus (2)/(3) for operators who need to tune the datapath on multi-NIC hosts.

## Impact

Any host with an `ixgbe`-class NIC on its public/default-route interface loses public connectivity immediately after `kube cluster install`, with no provisioner-level remediation. Recovery currently requires out-of-band console access to detach XDP (`ip link set dev xdp off`) or to reconfigure Cilium by hand.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.