kubernetes-sigs / kubernetes-sigs/kernel-module-management

Support pre-unload hooks for modules with active PCI device bindings

Open
#1,324 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
130
Forks
44
Avg merge
5d 16h
Merged PRs (30d)
8

Description

KMM's module unloader uses `modprobe -rv` to remove in-tree modules before loading OOT replacements. This fails when PCI devices are bound to the module, because device bindings hold a non-zero refcount independent of any Kubernetes workload or device plugin state.

## Current behavior

The ordered upgrade mechanism sequences: device plugin removal → module unload → module load → device plugin restart. This assumes `modprobe -r` will succeed once Kubernetes device plugins are stopped. However, PCI device bindings persist independently of Kubernetes, as the kernel holds a refcount for each device bound to the driver via `*_pci_probe()`.

For example, with the `xe` (Intel GPU) driver and SR-IOV virtual functions active:

```
$ cat /sys/module/xe/refcnt
4 # 2 PFs + VFs bound via udev
```

`modprobe -rv xe` fails with `EBUSY`. The only workaround is manual device teardown before KMM can act:

```bash
# zero VFs
echo 0 > /sys/class/drm/card*/device/sriov_numvfs
# unbind PFs
echo 0000:XX:YY.0 > /sys/bus/pci/drivers/xe/unbind
```

The same issue applies on ClusterPolicy deletion, as KMM's unloader cannot remove the OOT module while devices are bound.

## Impact

This affects any kernel module whose driver binds PCI (or other bus) devices, not just `xe`. Common scenarios:

- GPU drivers with SR-IOV VFs (xe, i915, nvidia)
- Network drivers with active VFs
- Any driver where udev auto-binds devices on module load

KMM's `inTreeModulesToRemove` is the primary mechanism for OOT driver replacement, so this is a significant gap for the OOT use case on systems with active device bindings.

## Proposed change

Add support for a pre-unload hook in the Module CR, i.e., a container or script that runs before `modprobe -rv`, allowing operators to perform device teardown (unbind PCI devices, zero VFs, etc.) specific to their hardware. A symmetric post-load hook would handle device re-setup after the OOT module is loaded.

Something like:

```yaml
spec:
moduleLoader:
container:
modprobe:
moduleName: xe
preUnload:
command: ["/bin/sh", "-c", "..."] # unbind devices, zero VFs
postLoad:
command: ["/bin/sh", "-c", "..."] # recreate VFs, rebind
```

This would keep device-specific teardown logic with the operator/user rather than requiring KMM to understand PCI topology, while giving KMM the extension point needed to handle modules with active device bindings.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Module CR's moduleLoader.modprobe configuration and the ordered upgrade sequence to find where modprobe -rv runs. Check the ClusterPolicy deletion path as well; done means pre-unload and post-load commands run around module replacement so operators can tear down and restore device bindings.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.