enhancement: A recreated Devices ledger never restores the node's accelerator counting capacity

What would you like to be added:

Make NodeCapacityReconciler recover a node's accelerator counting capacity when its Devices
ledger is recreated. Today the recreation is silently ignored, and the node keeps zero counting
capacity until the worker is restarted.

Concretely, the Devices watch at
pkg/worker/controllers/worker/node_capacity.go#L500-L533
needs to enqueue the node on the update that turns an unmanaged Devices into a managed one, not
only on an accelerator-detail change.

Why is this needed:

Note: this reads as a defect rather than a feature; it is filed as an enhancement per the reporter's
preference. It is pre-existing, present on main, and independent of any one manufacturer.

Symptom

Delete a node's Devices object. The node's owned accelerator counting keys are removed (correct —
the ledger is the source of truth). The Devices is then recreated by the device manager on its
next detect start-up, complete and correct — but the keys never come back. The node stays
unschedulable for every accelerator request until gpustack-operator-worker is restarted.

Measured on a single-node cluster, all timings from one uninterrupted trace:

t	action	observed
0s	kubectl delete devices <node>	—
+5s	—	worker logs enqueued node from devices, then patched node accelerator capacity; every owned key (<vendor>.com/<family>.sliced.*, .partitioned.<profile>, .units) removed. The bare device-plugin pool is untouched, as designed.
+150s	—	Devices still absent — nothing recreates it on its own.
+155s	kubectl -n gpustack-system rollout restart ds/<device-manager>	Devices recreated within 5s: generation: 1, full spec.groups, carrying gpustack.ai/managed: "true".
+155s … +245s	—	keys still absent, and the worker logs no enqueued node from devices at all.
+245s	kubectl -n gpustack-system rollout restart deploy/gpustack-operator-worker	all keys restored immediately.

A device-manager restart on its own — with the Devices object left alone — does not lose the
keys. A 124-second sampling run across a rollout restart of the DaemonSet showed every key
unchanged throughout. The trigger is specifically the deletion and recreation of the ledger.

Root cause

Two predicates that are individually reasonable combine into a permanent miss. The Devices object
is created without the gpustack.ai/managed label, by design:

// pkg/devicemanager/detector/detector.go#L466-L471
// Stamp the accelerator flavors' selector labels (os/arch + feature key) so the worker
// locates this node's Devices by one List. ...
// gpustack.ai/managed is synced separately by NodeDevicesReconciler.
devsLabels := acceleratableDevicesSelectorLabels(nd, aNf.Spec.Labels)

The label arrives afterwards, in a separate Update issued by NodeDevicesReconciler
(node_devices.go#L84-L90),
which mirrors it from the Node. So the sequence a recreation produces is:

1. Create — object has the selector labels and the full spec.groups, but no
   gpustack.ai/managed.
   → CreateFunc is isManagedDevices(e.Object)
   (node_capacity.go#L519-L521)
   → false. Dropped.

2. Update — NodeDevicesReconciler adds gpustack.ai/managed: "true". Nothing else moves.
   → UpdateFunc passes isManagedDevices(newDevs) but then gates on
   acceleratorDetailChanged(oldDevs, newDevs)
   (node_capacity.go#L525-L531).
   That function compares only spec.groups and the status ledger — labels are deliberately outside
   the signature
   (node_capacity.go#L553-L612)
   → false. Dropped.

3. Nothing else on the Devices changes, and the Node itself is already settled, so its own
   For(&core.Node{}) watch never fires either. The node is never enqueued again.

The removal in step 1 of the trace is correct and should stay: Reconcile sets devs = nil on
NotFound (node_capacity.go#L81-L89),
desiredAcceleratorCapacity returns an empty set, and buildAcceleratorCapacityPatch reverse-patches
the stale keys away. With no ledger there is nothing to advertise. The bug is only that the return
of the ledger is not observed.

This is why a fresh install is unaffected. On a new node the ledger is created while the node is
still churning — NFD is stamping feature labels, the device plugin is registering its extended
resources — and any of those Node events enqueues the node after the Devices is complete. A
delete-and-recreate on a settled node has no such follow-up event.

Two things ruled out along the way, so nobody re-derives them:

• Not the dedup window. DedupEnqueueRequestsFromMapFuncWithWindow defers by at most one TTL
  (3s) and expires entries at 2*ttl; it cannot swallow an event for 90 seconds.
• Not a stale informer. The Delete event was delivered and logged. The Create is delivered
  too — it is filtered by the predicate, not lost.

Scope

NodeCapacityReconciler is the only controller with this combination. InstanceTypeReconciler
watches Devices with a plain NewPredicateFuncs
(instance_type.go#L969-L975),
which is applied to the new object on update, so the label-adding update passes it and the
four-view recovers on its own.

Every manufacturer is affected identically — the dropped keys are the family/.sliced.*/
.partitioned.*/.units set, all of which the worker owns. The bare device-plugin pool
(<vendor>.com/<family>) is never touched by this controller and stays correct throughout, which is
what makes the state hard to spot: the node still looks like it has accelerators.

Suggested fix

Make the update predicate fire when the managed mark itself appears, not only when accelerator
detail moves:

UpdateFunc: func(e ctrlevent.UpdateEvent) bool {
    oldDevs, newDevs := e.ObjectOld.(*workercore.Devices), e.ObjectNew.(*workercore.Devices)
    if !isManagedDevices(newDevs) {
        return false
    }
    // A ledger becomes managed after it is created: the DeviceManager creates it
    // unlabelled and NodeDevicesReconciler mirrors gpustack.ai/managed from the Node
    // afterwards. That update carries no accelerator-detail change, so it must be
    // admitted on the transition itself or a recreated ledger is never observed.
    if !isManagedDevices(oldDevs) {
        return true
    }
    return acceleratorDetailChanged(oldDevs, newDevs)
},

An alternative worth weighing is having the device manager stamp gpustack.ai/managed at creation
time; it was deliberately not done — the comment on NodeDevicesReconciler states that node
management is a control-plane decision a per-node device manager must not assert — so the predicate
is the smaller and more faithful change.

Workaround

$ kubectl -n gpustack-system rollout restart deploy/gpustack-operator-worker

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

A unit test on the predicate is the natural regression guard: an unmanaged → managed update with
identical spec.groups must enqueue.

Start in pkg/worker/controllers/worker/node_capacity.go at the Devices watch and read the managed-label handling alongside acceleratorDetailChanged. Check the creation and label-update flow in detector.go and node_devices.go, then add a regression unit test for an unmanaged-to-managed update with identical accelerator details. Done means that transition enqueues the node without changing existing detail-change behavior.