gpustack / gpustack/gpustack-operator

todo: observe the slice from inside a container on Iluvatar and MThreads

Open
#138 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/vendor/iluvatar area/vendor/mthreads todo
Dominant language
Go
Stars
4
Forks
7
Avg merge
3h 9m
Merged PRs (30d)
213

Description

What was left undone

device-manager preflight answers three questions at three depths. For Iluvatar and MThreads the
second question — can these accelerators actually be sliced — stops at simulated on every row,
saying so in its own words:

capability: sliced-runtime-loaded
state: ok
depth: simulated
detail: the allocator produced the slice injection; no container probe has been established
  for iluvatar, so what the slice looks like from inside it was not observed

That is an honest answer, not a gap in the runner: the allocator really did produce an injection, and
that much was established. What is missing is the other half — starting a container with that
injection and reading back what the slice looks like from inside it.

Preflight calls this the injection-only tier, documented in
docs/operation/preflight.md (What you get, by manufacturer).

Where it came from

PR #136, which built the command and its nine per-manufacturer preflighters. sliceProbes in
pkg/devicemanager/preflight/measure.go carries only the manufacturers whose slice has been observed
from inside a container, and deliberately carries nothing for these two — the table's own comment
states that it holds only what has been measured, never a guess per manufacturer.

Hygon has since left this tier, and is the worked precedent. PR #140 ran it on an 8-DCU host and
established all three of the unproven properties below, moving both its rows to measured. What that
took, and what it disproved on the way, is recorded in
specs/2026-08-30-hygon-dcu-sliced-allocation.md and in the comment on this
issue
. Follow its
shape rather than re-deriving it.

What it needs in order to run

One card of each manufacturer, on a host with a container runtime. Neither is reachable today.

How to verify it

Per manufacturer, in this order — the cases are the prerequisite for the table, not a follow-up to it.

  1. Reproduce the injection by hand on the hardware, and add the numbered cases under
    .claude/skills/gpustack-operator-xbuild-and-verify/cases/. That skill is where every value in
    sliceProbes was measured. The five Hygon cases are the shape to copy: record rendering and mount
    shape (often no card required), single-card injection read back from inside a container, the
    memory cap actually enforced rather than reported, the compute limit taking effect, and two slices
    on one card staying independent.
  2. Add the manufacturer to sliceProbes using only what those cases measured, and confirm the
    rows move from simulated to measured on that host.
  3. Update the tier table in docs/operation/preflight.md and the per-manufacturer table beside it.

The two are not symmetric, and the harder one is not the obvious one

  • Iluvatar is the NVIDIA shape. It preloads HAMi-core's libvgpu.so through /etc/ld.so.preload
    and carries LIBCUDA_LOG_LEVEL and CUDA_DEVICE_MEMORY_LIMIT_*. The existing mapped-object test
    and MemoryQuotaEnvPrefix should serve it directly; the obstacles Hygon hit were specific to a
    vendor-owned slicing runtime and do not apply here.

  • MThreads is the Hygon shape. Its sliced injection is environment variables only —
    MTHREADS_QOS_MEMORY_LIMIT, MTHREADS_QOS_COMPUTING_POWER_WEIGHT, MTHREADS_VISIBLE_DEVICES
    enforced by the host sGPU kmod and the vendor container runtime. Nothing of ours is mounted, so
    there is no shared object to find in a container's address space and no log level of ours to raise.
    It will need a sliceProbe.LoadEvidence of its own: a string that exists only inside a slice, which
    the reader can cause and then read.

    Hygon's lesson about where that string is not likely to come from is worth carrying: its vendor
    SMI tool answers from the DMI layer and reports the physical card under a fully capped container, so
    a probe built on mthreads-gmi should be assumed to do the same until measured otherwise.

A note on scope, already settled

The gpustack-operator-xbuild-and-verify skill is a build-and-verify skill: its backends each
build an artifact this repository owns, then verify it. Neither of these two has such an artifact —
csrc/ carries only amd and thead. Hygon reached the same fork and was resolved by adding cases
without a backend
, and SKILL.md now carries that as a rule. Follow it; do not add a backend with no
build step.

What is unproven until then

For these two: whether the vendor runtime honours the injection at all, whether the memory cap is
enforced rather than merely declared, and whether two slices on one card stay isolated. The allocator
renders an injection that no test has ever watched a runtime read.

They are not entirely untouched, though: since #140, an injection carrying nothing is a failure on
the no-probe branch, where an empty ContainerAllocateResponse used to come back ok. That branch is
now the whole of what these two get.

Related

  • #96 needs step 1 above for each of these two, for the same reason it needed it for Hygon: its first
    verification step is to create a logically sliced Instance, which cannot mean anything until the
    slice is known to take effect.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pkg/devicemanager/preflight/measure.go and the existing Hygon precedent in specs/2026-08-30-hygon-dcu-sliced-allocation.md. Run the numbered hardware cases under .claude/skills/gpustack-operator-xbuild-and-verify/cases/ for each manufacturer, then update sliceProbes and the tables in docs/operation/preflight.md. Done means measured rows backed by container observations, including enforced limits and independent slices.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
documentation, infrastructure, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.