ParseMigProfile returns IDs from wrong GPU on heterogeneous hosts
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 58
- Forks
- 34
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 7
Description
Summary
ParseMigProfile returns NVML profile IDs from the wrong GPU on heterogeneous hosts because VisitMigProfiles dedupes profiles by name across all GPUs.
What breaks
- On hosts where the same MIG profile name (e.g.
"1g.10gb") maps to different NVMLGIProfileIDs on different GPUs, only the first-walked variant survives. - Callers using
(*devicelib).ParseMigProfileto drive NVML calls on a specific GPU may issue requests with a profile ID that GPU does not support. - Symptom:
GetGpuInstanceProfileInforeturnsERROR_NOT_SUPPORTED, and operations likeSetMigConfigfail.
Root cause
(*devicelib).VisitMigProfiles builds a map[string]struct{} keyed by p.String(), so cross-device duplicates are silently dropped (device.go#L537-L562):
visited := make(map[string]struct{})
err := d.VisitDevices(func(i int, d Device) error {
return d.VisitMigProfiles(func(p MigProfile) error {
if _, ok := visited[p.String()]; ok { // keyed by name — the bug
return nil
}
visited[p.String()] = struct{}{}
return visit(p)
})
})
GetMigProfiles caches that deduped list on devicelib.migProfiles (device.go#L590-L610), and ParseMigProfile searches it by name (mig_profile.go#L152-L166). The per-device (*device).VisitMigProfiles does not dedupe (device.go#L352-L431).
All refs against main at commit 8ff29bb.
Failure example
GPU 0 exposes "1g.10gb" via GPU_INSTANCE_PROFILE_1_SLICE_REV2; GPU 1 exposes it via GPU_INSTANCE_PROFILE_1_SLICE. Then SetMigConfig(gpu=1, config={"1g.10gb": 1}):
ParseMigProfile("1g.10gb")returns the GPU 0 variant (REV2), because GPU 0 was walked first.- On GPU 1,
GetGpuInstanceProfileInfo(REV2)returnsERROR_NOT_SUPPORTED. SetMigConfigfails and leaves GPU 1 cleared.
Reproduction / workaround
Reproduced and worked around in mig-parted PR #372, which adds a per-device resolver plus a test (TestSetMigConfigResolvesMigProfileOnTargetGPU in pkg/mig/config/config_test.go) that mocks the heterogeneous case. Originating report: mig-parted#157.
Consumer-side workaround: resolve profile names against the per-device Device.GetMigProfiles() rather than the devicelib cache.
Possible directions
- A. Drop the dedup in
(*devicelib).VisitMigProfiles; return one entry per(GPU, name). Changes the observable shape ofGetMigProfiles/ParseMigProfile. - B. Keep the dedup, but have
ParseMigProfilereturn a typedErrAmbiguouswhen devices disagree on GI/CI IDs for a name. - C. Document the current behavior as "device-agnostic, first-walked wins" and steer NVML-bound callers to the per-device
Device.GetMigProfiles().
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with VisitMigProfiles and GetMigProfiles in pkg/nvlib/device/device.go, then read ParseMigProfile in pkg/nvlib/device/mig_profile.go and compare the per-device traversal. Reproduce the heterogeneous-profile case described in the issue and inspect TestSetMigConfigResolvesMigProfileOnTargetGPU for expected consumer behavior. The issue is complete only after a direction among A–C is selected and covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100