gpustack / gpustack/gpustack-operator
todo: verify the T-Head partition process check and the utilization classification on hardware
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4
- Forks
- 7
- Avg merge
- 3h 9m
- Merged PRs (30d)
- 213
Description
What was left undone
Running the two T-Head halves of this change against a PPU driver. Both are mirrors of the NVIDIA
halves, and the NVIDIA ones were verified on an H100 — but no HGML call in either is exercised by
anything except a fake driver.
- The reclaim process check.
InstanceProcessesresolves the partition-device handle by
GPU-instance id and counts its compute processes; the reclaim loop asks it before every destroy in
both directions and keeps the partition, and its ownership marker, when the answer is not zero. - The utilization classification. A whole-device cores-utilization read answering
NOT_SUPPORTED
is now reported at the detector's own verbosity instead of as a per-round error, on the grounds that
a partitioning accelerator accounts compute per partition and has no whole-device aggregate.
Where it came from
This branch's fix(allocator): never reclaim a mig partition a process is running on and
fix(detector): stop reporting a partitioning device's missing aggregate as an error. The T-Head
surface is pkg/devicemanager/allocator/thead/mig_driver_linux.go (InstanceProcesses),
pkg/devicemanager/allocator/thead/mig_reclaim.go (processRunningOn, called from both destroy
paths), binding/hgml/library_device.go (GetComputeRunningProcesses added to the MigDevice
interface), and pkg/devicemanager/detector/thead/device.go (coresUtilizationOf). Covered by three
table cases in the reclaim suite — two of which fail without the guard — and by nothing else.
What it needs in order to run
A T-Head PPU host with MIG mode available, reachable as a Kubernetes node. The two known constraints
from earlier PPU work still apply: toggling -mig 1 fails EBUSY while any process holds the
driver on any card, and a PPU partition pod needs no runtimeClassName (unlike NVIDIA/MThreads).
Step 2 also needs an image able to hold a compute context on a PPU partition.
How to verify it
- Enable MIG on one PPU, restart the device-manager DaemonSet (a mode toggle fires no re-detect), and
wait for thealibabacloud.com/ppu.partitioned.mig-*node keys. - Carve a partition out of band on an otherwise drained PPU and hold a compute context on it.
Across more thanreclaimMaxMissespasses assert: no destroy is attempted, no error line appears,
and at-v=3the loop saysreclaim: a process is running on this partition, skipping destroy
with the partition's GPU-instance id and the process count. - Release the process. The next pass must reclaim that partition, which is what returns its placement
to the pool. - Make the query fail — a partition the enumeration offers no handle for is enough — and confirm the
destroy still proceeds. Failing open there is deliberate: reading "cannot ask" as "in use"
would stop reclamation entirely wherever the query fails, so the driver's own busy refusal is what
guards that case. On NVIDIA that refusal is measured; on T-Head it is not. - For the detector half: with MIG on,
kubectl logs … | grep -c "cores utilization"across at least
20 monitor rounds at-v=2must be 0, and the published reading for that card is the
deliberate zero (this figure has no absent form — a partition's own entry reports absence
properly).
What is unproven until then
Two driver answers, both assumed from the NVML analogue:
- whether HGML answers
GetComputeRunningProcesseson a PPU partition handle at all. If it does
not, every reclaim falls to the fail-open branch and the behaviour is exactly today's — no
regression, but no new guarantee either, and the log line at-v=3is how you would tell. - whether HGML's
GetUtilizationRatesanswersNOT_SUPPORTEDon a partitioning PPU. If it answers
something else, the classification falls to the default error branch and the per-round noise stays.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pkg/devicemanager/allocator/thead/mig_driver_linux.go, mig_reclaim.go, binding/hgml/library_device.go, and pkg/devicemanager/detector/thead/device.go; read InstanceProcesses, processRunningOn, GetComputeRunningProcesses, and coresUtilizationOf. Run the reclaim suite's three table cases and test on a T-Head PPU host with MIG enabled. Done means the process-held partition is preserved, query failures still allow destruction, and unsupported aggregate utilization produces no per-round error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100