enhancement: The sliced working-dir GC deletes live partition ownership records, and reclaim then destroys the partitions

What would you like to be added:

Stop the logical-slicing working-directory GC from deleting the hardware-partition ownership
records that live in the same per-Pod tree. Today it removes the whole <OperatorPodsDir>/<podUID>
directory, and a partition whose record is gone is indistinguishable from an abandoned one — so the
vendor reclaim loop destroys it while its Pod is alive.

Note: this reads as a defect rather than a feature; it is filed as an enhancement per the reporter's
preference. It is pre-existing on main and affects every partitioning manufacturer — NVIDIA,
T-Head and Hygon alike.

Why is this needed:

The two surfaces share one directory tree

A hardware partition's ownership record is written under the per-Pod tree:

Manufacturer	Record path
NVIDIA	deviceplugin.OperatorPodsDir/<podUID>/c-<container>/… (nvidia/mig.go#L622)
T-Head	same tree (thead/mig_visibility.go#L101)
Hygon	same tree (hygon/mig.go, migMarkerPath)

The logical-slicing server owns a GC over that same root:

// pkg/deviceplugin/server.go#L117-L119
var gc *podDirGC
if s.AllocationMode == workercore.DeviceAllocationModeSliced {
    gc = newPodDirGC(OperatorPodsDir)
}

and it reclaims by removing the whole Pod directory:

// pkg/deviceplugin/gc.go#L64
if err := os.RemoveAll(filepath.Join(g.dir, uid)); err == nil {

The Sliced server runs even where no logical slice can ever be served. It is constructed on
!opts.NoSliced alone, independent of whether the node's cards advertise any sliced capability — so
on a Hygon node in MIG mode, where hygon.com/dcu.sliced is 0, the GC is running over the tree
that holds every live partition's record.

The failure

podDirGC.reconcile is driven per broadcast from notifyListeners, which fires on every reconcile
and synchronously from reserveDevices/releaseReservation. A freshly admitted Pod is not in
lastLivePodUIDs until the reconcile that observes its allocation annotation, so a burst of
admissions can deliver several broadcasts in a row that do not name it. Three consecutive misses is
the threshold (podDirGCMaxMisses = 3), and at broadcast rate that is well under a second.

Once the record is gone:

1. scanMigMarkers finds nothing for that Pod, so the partition is in no seen set.
2. The grace window does not help. It guards the record-names-a-dead-Pod path only; a partition
   with no record at all goes to the orphan sweep, which has no age to check.
3. destroyIfStillUnclaimed re-reads the records under the accelerator lock — and still finds
   nothing, because the record was deleted rather than merely unseen. It destroys the partition.
4. The driver refuses only if a process already holds the device. A container that has been granted
   but has not yet opened /dev/kfd is not protected.

The container then starts against a registry file that no longer exists, and the container runtime
creates a directory at that path — which permanently poisons that instance id, since
INSUFFICIENT_RESOURCES then follows the id forever. That secondary failure was observed on
hardware: two ids became permanently uncreatable on a Hygon node.

Why the existing mitigations do not cover it

0d0d1e5-era work added migReclaimGrace and the under-lock re-read specifically because a
stale scan was destroying just-granted partitions. Both fixes assume the record is on disk.
Neither can see a record that was deleted out from under them.

A second, weaker path to the same outcome

RunReclaimLoop's broadcast channel is deliberately lossy (buffered, non-blocking send), and the
60 s resync reuses lastLive from the most recent broadcast that was actually received
(reclaim.go#L44-L60).
Under sustained saturation lastLive can stay stale past the two-minute grace, at which point the
record-names-a-dead-Pod path destroys a live partition on its own. This needs continuous drops for
the whole grace window, so it is far less likely than the GC path above — but it has the same blast
radius and would be fixed in the same neighbourhood.

Suggested fix

Preferred: scope the GC to what it owns. The sliced artifacts live at a known depth under the Pod
directory (c-<container>/etc/vdev/..., .../nvidia/...), so the GC can remove those rather than the
Pod root, and remove the Pod root only when nothing else remains in it.

Cheaper alternative: have podDirGC skip any Pod directory that contains a partition ownership
record. That leaks an empty directory for a genuinely dead Pod's partition until the vendor reclaim
removes the record, which is the safe direction.

Either way the reclaim's own decision would be improved by treating "no record at all" as
inconclusive when the record tree could have been touched — but scoping the GC is what removes the
cause.

Provenance

Found by an independent cross-model review of the Hygon partitioning branch, then confirmed against
the source. Not introduced by that branch — the same tree, the same GC and the same reclaim shape
predate it on NVIDIA and T-Head.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

A unit test on podDirGC.reconcile is the natural regression guard: a Pod directory holding a
partition ownership record must survive podDirGCMaxMisses consecutive misses.

Start with pkg/deviceplugin/gc.go and server.go, then trace reclaim.go and the ownership paths in allocator/nvidia/mig.go, allocator/thead/mig_visibility.go, and allocator/hygon/mig.go. Add the unit regression around podDirGC.reconcile so a directory containing a partition record survives podDirGCMaxMisses misses. Done also requires the listed design doc, API change, and documentation update.