kubernetes-sigs / kubernetes-sigs/kernel-module-management
Support pre-unload hooks for modules with active PCI device bindings
- Dominant language
- Go
- Stars
- 130
- Forks
- 44
- Avg merge
- 5d 16h
- Merged PRs (30d)
- 8
Description
KMM's module unloader uses `modprobe -rv` to remove in-tree modules before loading OOT replacements. This fails when PCI devices are bound to the module, because device bindings hold a non-zero refcount independent of any Kubernetes workload or device plugin state.
## Current behavior
The ordered upgrade mechanism sequences: device plugin removal → module unload → module load → device plugin restart. This assumes `modprobe -r` will succeed once Kubernetes device plugins are stopped. However, PCI device bindings persist independently of Kubernetes, as the kernel holds a refcount for each device bound to the driver via `*_pci_probe()`.
For example, with the `xe` (Intel GPU) driver and SR-IOV virtual functions active:
```
$ cat /sys/module/xe/refcnt
4 # 2 PFs + VFs bound via udev
```
`modprobe -rv xe` fails with `EBUSY`. The only workaround is manual device teardown before KMM can act:
```bash
# zero VFs
echo 0 > /sys/class/drm/card*/device/sriov_numvfs
# unbind PFs
echo 0000:XX:YY.0 > /sys/bus/pci/drivers/xe/unbind
```
The same issue applies on ClusterPolicy deletion, as KMM's unloader cannot remove the OOT module while devices are bound.
## Impact
This affects any kernel module whose driver binds PCI (or other bus) devices, not just `xe`. Common scenarios:
- GPU drivers with SR-IOV VFs (xe, i915, nvidia)
- Network drivers with active VFs
- Any driver where udev auto-binds devices on module load
KMM's `inTreeModulesToRemove` is the primary mechanism for OOT driver replacement, so this is a significant gap for the OOT use case on systems with active device bindings.
## Proposed change
Add support for a pre-unload hook in the Module CR, i.e., a container or script that runs before `modprobe -rv`, allowing operators to perform device teardown (unbind PCI devices, zero VFs, etc.) specific to their hardware. A symmetric post-load hook would handle device re-setup after the OOT module is loaded.
Something like:
```yaml
spec:
moduleLoader:
container:
modprobe:
moduleName: xe
preUnload:
command: ["/bin/sh", "-c", "..."] # unbind devices, zero VFs
postLoad:
command: ["/bin/sh", "-c", "..."] # recreate VFs, rebind
```
This would keep device-specific teardown logic with the operator/user rather than requiring KMM to understand PCI topology, while giving KMM the extension point needed to handle modules with active device bindings.
Contributor guide
Research direction
Start by tracing the Module CR's moduleLoader.modprobe configuration and the ordered upgrade sequence to find where modprobe -rv runs. Check the ClusterPolicy deletion path as well; done means pre-unload and post-load commands run around module replacement so operators can tear down and restore device bindings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100