gpustack / gpustack/gpustack-operator
todo: observe the slice from inside a container on Iluvatar and MThreads
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4
- Forks
- 7
- Avg merge
- 3h 9m
- Merged PRs (30d)
- 213
Description
What was left undone
device-manager preflight answers three questions at three depths. For Iluvatar and MThreads the
second question — can these accelerators actually be sliced — stops at simulated on every row,
saying so in its own words:
capability: sliced-runtime-loaded
state: ok
depth: simulated
detail: the allocator produced the slice injection; no container probe has been established
for iluvatar, so what the slice looks like from inside it was not observed
That is an honest answer, not a gap in the runner: the allocator really did produce an injection, and
that much was established. What is missing is the other half — starting a container with that
injection and reading back what the slice looks like from inside it.
Preflight calls this the injection-only tier, documented in
docs/operation/preflight.md (What you get, by manufacturer).
Where it came from
PR #136, which built the command and its nine per-manufacturer preflighters. sliceProbes in
pkg/devicemanager/preflight/measure.go carries only the manufacturers whose slice has been observed
from inside a container, and deliberately carries nothing for these two — the table's own comment
states that it holds only what has been measured, never a guess per manufacturer.
Hygon has since left this tier, and is the worked precedent. PR #140 ran it on an 8-DCU host and
established all three of the unproven properties below, moving both its rows to measured. What that
took, and what it disproved on the way, is recorded in
specs/2026-08-30-hygon-dcu-sliced-allocation.md and in the comment on this
issue. Follow its
shape rather than re-deriving it.
What it needs in order to run
One card of each manufacturer, on a host with a container runtime. Neither is reachable today.
How to verify it
Per manufacturer, in this order — the cases are the prerequisite for the table, not a follow-up to it.
- Reproduce the injection by hand on the hardware, and add the numbered cases under
.claude/skills/gpustack-operator-xbuild-and-verify/cases/. That skill is where every value in
sliceProbeswas measured. The five Hygon cases are the shape to copy: record rendering and mount
shape (often no card required), single-card injection read back from inside a container, the
memory cap actually enforced rather than reported, the compute limit taking effect, and two slices
on one card staying independent. - Add the manufacturer to
sliceProbesusing only what those cases measured, and confirm the
rows move fromsimulatedtomeasuredon that host. - Update the tier table in
docs/operation/preflight.mdand the per-manufacturer table beside it.
The two are not symmetric, and the harder one is not the obvious one
-
Iluvatar is the NVIDIA shape. It preloads HAMi-core's
libvgpu.sothrough/etc/ld.so.preload
and carriesLIBCUDA_LOG_LEVELandCUDA_DEVICE_MEMORY_LIMIT_*. The existing mapped-object test
andMemoryQuotaEnvPrefixshould serve it directly; the obstacles Hygon hit were specific to a
vendor-owned slicing runtime and do not apply here. -
MThreads is the Hygon shape. Its sliced injection is environment variables only —
MTHREADS_QOS_MEMORY_LIMIT,MTHREADS_QOS_COMPUTING_POWER_WEIGHT,MTHREADS_VISIBLE_DEVICES—
enforced by the host sGPU kmod and the vendor container runtime. Nothing of ours is mounted, so
there is no shared object to find in a container's address space and no log level of ours to raise.
It will need asliceProbe.LoadEvidenceof its own: a string that exists only inside a slice, which
the reader can cause and then read.Hygon's lesson about where that string is not likely to come from is worth carrying: its vendor
SMI tool answers from the DMI layer and reports the physical card under a fully capped container, so
a probe built onmthreads-gmishould be assumed to do the same until measured otherwise.
A note on scope, already settled
The gpustack-operator-xbuild-and-verify skill is a build-and-verify skill: its backends each
build an artifact this repository owns, then verify it. Neither of these two has such an artifact —
csrc/ carries only amd and thead. Hygon reached the same fork and was resolved by adding cases
without a backend, and SKILL.md now carries that as a rule. Follow it; do not add a backend with no
build step.
What is unproven until then
For these two: whether the vendor runtime honours the injection at all, whether the memory cap is
enforced rather than merely declared, and whether two slices on one card stay isolated. The allocator
renders an injection that no test has ever watched a runtime read.
They are not entirely untouched, though: since #140, an injection carrying nothing is a failure on
the no-probe branch, where an empty ContainerAllocateResponse used to come back ok. That branch is
now the whole of what these two get.
Related
- #96 needs step 1 above for each of these two, for the same reason it needed it for Hygon: its first
verification step is to create a logically sliced Instance, which cannot mean anything until the
slice is known to take effect.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pkg/devicemanager/preflight/measure.go and the existing Hygon precedent in specs/2026-08-30-hygon-dcu-sliced-allocation.md. Run the numbered hardware cases under .claude/skills/gpustack-operator-xbuild-and-verify/cases/ for each manufacturer, then update sliceProbes and the tables in docs/operation/preflight.md. Done means measured rows backed by container observations, including enforced limits and independent slices.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- documentation, infrastructure, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 30/100