gpustack / gpustack/gpustack-operator
enhancement: A partition requested inside the previous instance's reclaim window is refused, not queued
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4
- Forks
- 7
- Avg merge
- 3h 9m
- Merged PRs (30d)
- 213
Description
Summary
Deleting a partition-backed Pod frees the card in the ledger immediately, but the hardware
instance itself is only destroyed after the reclaimer's debounce (three consecutive misses). In
between, the node advertises room the card does not yet have, so a replacement scheduled into that
window is admitted by the scheduler and then refused at allocation time — surfacing on the Pod as
UnexpectedAdmissionError.
A workload that retries converges by itself. A workload that does not — restartPolicy: Never with no
owning controller — simply fails.
Mechanism
Occupancy for placement comes from DevicesReconciler.LivePhysicalOccupied, which builds the
occupied set from the allocation annotations of live Pods on the node. That set is the right one
for accounting, but it is not the physical truth during reclaim:
- the Pod object is gone, so its placement leaves
LivePhysicalOccupied, and the ledger's
remainingplus the node's*.partitioned*keys go back to the idle values; - the vendor instance still exists, because
reclaimMaxMissespasses must observe it unowned first; - a replacement therefore picks a placement the card cannot actually host, and creating it fails.
Evidence
Reproduced on a single-card NVIDIA H100 with the partition e2e scenarios:
- The single-shot form fails. In the
allocated + remainingscenario, the refill Pod submitted
right after the saturating Pod was deleted never ran:Failed/UnexpectedAdmissionError. Everything
else in that scenario passed — the freed count was advertised correctly (1 → 7), the room just was
not usable yet. - The retrying form self-heals. In the reclaim-window scenario, a same-profile replacement
converged on its second attempt, 45 s after the predecessor was deleted, with exactly one
terminal allocation failure along the way.
Also reproduced four times earlier on T-Head PPU hardware, so this is the shared reclaim path, not
one vendor's driver.
Not the same as the stale-identity bug
A related failure — the replacement being handed the destroyed instance's identity when a
GPU-instance id was reused — is fixed on the thead-ppu-mig-partitioning branch by re-reading the
card's partitions under the lock before destroying. That makes the failure inside this window
clean (a refusal, never a dangling identity). It does not close the window.
Options considered
- Count the not-yet-destroyed instances as occupied — union
LivePhysicalOccupiedwith the
instances the durable markers still record but no live Pod owns. Placement then avoids them, which
removes the window entirely wherever another placement is free, and turns the single-card-full case
into an honest "no room" refusal (back-pressure the scheduler and Kueue can act on) rather than a
failed create. Cheapest correct change; does not make the full-card refill succeed any sooner. - Reclaim synchronously during allocation — when the chosen placement is held by an unowned
instance, destroy it in place and carry on. Closes the window completely, including the full-card
case. Costs a vendor round trip insideAllocate, and needs the ownership test to be exactly the
reclaimer's, or it risks destroying an instance whose Pod is still arriving. - Hold the ledger until the instance is really gone — release
remainingand the node keys on
destroy rather than on Pod deletion. Most truthful, and the scheduler stops being told about room
that does not exist; makes every release visibly slower. - Leave it to callers — document the window and rely on retries. What happens today.
Worth noting that (1) also addresses the other half of the observation: an instance carved outside
GPUStack is never reflected in the node keys at all, and that never converges.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at DevicesReconciler.LivePhysicalOccupied and trace how reclaimMaxMisses, ledger remaining, and partitioned node keys affect placement before Allocate. Reproduce the allocated + remaining and reclaim-window partition e2e scenarios, then choose and implement a documented approach from the issue's options. Done means replacements no longer reach an unusable reclaimed instance, with tests covering the reclaim window and full-card behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100