gpustack / gpustack/gpustack-operator

todo: verify the new T-Head PPU partition reader on hardware

Open
#94 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/vendor/thead todo
Dominant language
Go
Stars
4
Forks
7
Avg merge
3h 9m
Merged PRs (30d)
213

Description

What was left undone

Running the T-Head PPU partition reader against a driver. It is entirely new in #92 — before it, a
partition-backed Instance on a PPU fell through to the per-process pass and reported the parent card,
i.e. every tenant's memory under one tenant's name. The reader has never executed on hardware.

Two things need reading:

  • A PPU partition Instance reporting its own identifier, capacity and usage on all three surfaces,
    the way NVIDIA MIG now does.
  • The deliberate limitation: an allocation carrying no recorded identifier is answered as an absence
    with a reason. T-Head has no derivation fallback on purpose — HGML enumerates 85 GPU-instance profile
    ids where NVML enumerates 17, so translating a recorded profile name back into a profile id would
    cost hundreds of driver calls per card per monitor period.

Where it came from

PR #92, pkg/devicemanager/detector/thead/mig_process.go (new file), commit
feat(devicemanager): report a hardware partition on its own handle. Covered by 6 table cases plus a
driver-call-count pin, and by nothing else.

What it needs in order to run

A T-Head PPU host with MIG mode available, reachable as a Kubernetes node. Note the two known
constraints from earlier PPU work: toggling -mig 1 fails EBUSY while any process holds the
driver on any card, and a PPU slice pod needs no runtimeClassName (unlike NVIDIA/MThreads).

How to verify it

  1. Enable MIG on one PPU, restart the device-manager DaemonSet (a mode toggle fires no re-detect), and
    wait for the alibabacloud.com/ppu.partitioned.mig-* node keys.
  2. Create a partition Instance and read the subresource, the exporter and /monitor/snapshot. Assert:
    id is the partition's own UUID and not the parent card's; memoryTotalMiB is the partition's
    own capacity read off its handle; coresUtilizationPercent is absent with coresReason: unsupported.
  3. Confirm the parent card's figures appear nowhere in that entry except temperature, power and
    health — which every mode takes from the card by design.
  4. cases/case-39.sh for the surrounding PPU claims (pinned placement, a slice capped inside the
    container), which also has not run since this branch.
  5. Optionally confirm the no-identifier path by replaying a Pod annotated before the field: the entry
    must report an absence with a reason, not the card.

What is unproven until then

That the T-Head partition path produces any correct figure at all. This is the least-verified code in
#92 and the PR says so. The shared half of the path is not in question — partitionTargetsOf,
withPartitions, Figures and Resolve are manufacturer-agnostic and were exercised on an H100 — but
every HGML call in the new file is unexercised, including whether MigDevice.GetMemoryInfoV() answers
on a PPU partition handle at all.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read pkg/devicemanager/detector/thead/mig_process.go and run the verification on a T-Head PPU Kubernetes node with MIG enabled. Restart the device-manager DaemonSet, inspect the partition subresource, exporter, and /monitor/snapshot, and run cases/case-39.sh for surrounding claims. Done means the partition reports its own UUID, capacity, usage, and unsupported cores reason without inheriting parent-card figures; also check the no-identifier path if possible.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure, observability, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.