gpustack / gpustack/gpustack-operator

todo: verify the T-Head partition process check and the utilization classification on hardware

Open
#116 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/vendor/thead todo
Dominant language
Go
Stars
4
Forks
7
Avg merge
3h 9m
Merged PRs (30d)
213

Description

What was left undone

Running the two T-Head halves of this change against a PPU driver. Both are mirrors of the NVIDIA
halves, and the NVIDIA ones were verified on an H100 — but no HGML call in either is exercised by
anything except a fake driver.

  • The reclaim process check. InstanceProcesses resolves the partition-device handle by
    GPU-instance id and counts its compute processes; the reclaim loop asks it before every destroy in
    both directions and keeps the partition, and its ownership marker, when the answer is not zero.
  • The utilization classification. A whole-device cores-utilization read answering NOT_SUPPORTED
    is now reported at the detector's own verbosity instead of as a per-round error, on the grounds that
    a partitioning accelerator accounts compute per partition and has no whole-device aggregate.

Where it came from

This branch's fix(allocator): never reclaim a mig partition a process is running on and
fix(detector): stop reporting a partitioning device's missing aggregate as an error. The T-Head
surface is pkg/devicemanager/allocator/thead/mig_driver_linux.go (InstanceProcesses),
pkg/devicemanager/allocator/thead/mig_reclaim.go (processRunningOn, called from both destroy
paths), binding/hgml/library_device.go (GetComputeRunningProcesses added to the MigDevice
interface), and pkg/devicemanager/detector/thead/device.go (coresUtilizationOf). Covered by three
table cases in the reclaim suite — two of which fail without the guard — and by nothing else.

What it needs in order to run

A T-Head PPU host with MIG mode available, reachable as a Kubernetes node. The two known constraints
from earlier PPU work still apply: toggling -mig 1 fails EBUSY while any process holds the
driver on any card, and a PPU partition pod needs no runtimeClassName (unlike NVIDIA/MThreads).
Step 2 also needs an image able to hold a compute context on a PPU partition.

How to verify it

  1. Enable MIG on one PPU, restart the device-manager DaemonSet (a mode toggle fires no re-detect), and
    wait for the alibabacloud.com/ppu.partitioned.mig-* node keys.
  2. Carve a partition out of band on an otherwise drained PPU and hold a compute context on it.
    Across more than reclaimMaxMisses passes assert: no destroy is attempted, no error line appears,
    and at -v=3 the loop says reclaim: a process is running on this partition, skipping destroy
    with the partition's GPU-instance id and the process count.
  3. Release the process. The next pass must reclaim that partition, which is what returns its placement
    to the pool.
  4. Make the query fail — a partition the enumeration offers no handle for is enough — and confirm the
    destroy still proceeds. Failing open there is deliberate: reading "cannot ask" as "in use"
    would stop reclamation entirely wherever the query fails, so the driver's own busy refusal is what
    guards that case. On NVIDIA that refusal is measured; on T-Head it is not.
  5. For the detector half: with MIG on, kubectl logs … | grep -c "cores utilization" across at least
    20 monitor rounds at -v=2 must be 0, and the published reading for that card is the
    deliberate zero (this figure has no absent form — a partition's own entry reports absence
    properly).

What is unproven until then

Two driver answers, both assumed from the NVML analogue:

  • whether HGML answers GetComputeRunningProcesses on a PPU partition handle at all. If it does
    not, every reclaim falls to the fail-open branch and the behaviour is exactly today's — no
    regression, but no new guarantee either, and the log line at -v=3 is how you would tell.
  • whether HGML's GetUtilizationRates answers NOT_SUPPORTED on a partitioning PPU. If it answers
    something else, the classification falls to the default error branch and the per-round noise stays.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pkg/devicemanager/allocator/thead/mig_driver_linux.go, mig_reclaim.go, binding/hgml/library_device.go, and pkg/devicemanager/detector/thead/device.go; read InstanceProcesses, processRunningOn, GetComputeRunningProcesses, and coresUtilizationOf. Run the reclaim suite's three table cases and test on a T-Head PPU host with MIG enabled. Done means the process-held partition is preserved, query failures still allow destruction, and unsupported aggregate utilization produces no per-round error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.