Transient modprobe openvswitch failure permanently wedges kubelet/crio on boot, no self-recovery even after ovs-vswitchd recovers

Summary

A transient modprobe openvswitch failure during boot permanently prevents kubelet/crio from starting, with no automatic recovery, even though the underlying ovs-vswitchd.service itself self-heals via its own systemd restart policy a second later. Confirmed live on a control plane node during a mandatory MachineConfig rollout.

Reproduction, observed live

At boot, modprobe openvswitch failed transiently:

modprobe: ERROR: could not insert 'openvswitch': Cannot allocate memory

ovs-vswitchd.service's first start attempt failed as a direct result. Systemd's own Restart= policy successfully retried ovs-vswitchd.service about one second later, and it came up fine, confirmed active (running) and healthy for over an hour afterward.

However, openvswitch.service and ovs-configuration.service (templates/common/_base/units/ovs-configuration.service.yaml) had already evaluated their dependency on ovs-vswitchd.service at the moment of that first failure, and were marked Dependency failed, going inactive (dead) permanently with no further retry. This cascaded through nmstate-configuration.service to kubelet-dependencies.target (templates/common/_base/units/kubelet-dependencies.target.yaml) to crio.service and kubelet.service, all left inactive (dead) with result dependency. The node stayed NotReady for over an hour with zero self-recovery until manual intervention.

Recovery tested and confirmed

Once ovs-vswitchd.service was confirmed healthy again on its own, restarting ovs-configuration.service manually succeeded, but did not cascade to kubelet/crio automatically, those had to be explicitly restarted too:

systemctl restart ovs-configuration.service
systemctl restart kubelet-dependencies.target crio.service kubelet.service

Immediately after, the node returned to Ready. No further reboot was needed.

Why this matters

This happened during a routine MachineConfig rollout on a control plane node, not an unusual or contrived scenario. If it hit two control plane nodes at once (plausible if the underlying resource-pressure trigger for the transient modprobe failure is host-wide, for example a hypervisor host under memory pressure during a synchronized multi-VM reboot), it would risk actual etcd quorum loss during a disruptive rollout window. There is currently no automated recovery, and no indication anywhere in cluster status (ClusterOperators, MachineConfigPool) that points at the actual cause, a customer or support engineer would only see a node stuck NotReady and would need to SSH in, read raw journalctl boot logs, and understand systemd dependency/target semantics to find the real cause.

Recommendation

The dependent units in this boot chain (openvswitch.service, ovs-configuration.service, and by extension kubelet-dependencies.target) should re-evaluate or retry once the underlying ovs-vswitchd.service dependency recovers via its own restart policy, rather than staying permanently wedged from a single evaluation at the moment of first failure. This could be addressed with BindsTo=/OnFailure= semantics or an explicit retry/reset mechanism in the relevant unit files.

Environment

• RHCOS 10.2.20260902-0, OCP 5.0.0-rc.1

Filed from hackathon testing under OCPSTRAT-2677.

Start with templates/common/_base/units/ovs-configuration.service.yaml and templates/common/_base/units/kubelet-dependencies.target.yaml, then trace the systemd dependency chain through openvswitch.service, nmstate-configuration.service, crio.service, and kubelet.service. Reproduce or inspect the transient ovs-vswitchd failure and its later recovery. Done means the dependent units recover automatically and the node returns to Ready without manual service restarts.