NVIDIA / NVIDIA/go-nvlib

ParseMigProfile returns IDs from wrong GPU on heterogeneous hosts

Open
#91 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

lifecycle/stale
Dominant language
Go
Stars
58
Forks
34
Avg merge
3d 12h
Merged PRs (30d)
7

Description

Summary

ParseMigProfile returns NVML profile IDs from the wrong GPU on heterogeneous hosts because VisitMigProfiles dedupes profiles by name across all GPUs.

What breaks

  • On hosts where the same MIG profile name (e.g. "1g.10gb") maps to different NVML GIProfileIDs on different GPUs, only the first-walked variant survives.
  • Callers using (*devicelib).ParseMigProfile to drive NVML calls on a specific GPU may issue requests with a profile ID that GPU does not support.
  • Symptom: GetGpuInstanceProfileInfo returns ERROR_NOT_SUPPORTED, and operations like SetMigConfig fail.

Root cause

(*devicelib).VisitMigProfiles builds a map[string]struct{} keyed by p.String(), so cross-device duplicates are silently dropped (device.go#L537-L562):

visited := make(map[string]struct{})
err := d.VisitDevices(func(i int, d Device) error {
    return d.VisitMigProfiles(func(p MigProfile) error {
        if _, ok := visited[p.String()]; ok { // keyed by name — the bug
            return nil
        }
        visited[p.String()] = struct{}{}
        return visit(p)
    })
})

GetMigProfiles caches that deduped list on devicelib.migProfiles (device.go#L590-L610), and ParseMigProfile searches it by name (mig_profile.go#L152-L166). The per-device (*device).VisitMigProfiles does not dedupe (device.go#L352-L431).

All refs against main at commit 8ff29bb.

Failure example

GPU 0 exposes "1g.10gb" via GPU_INSTANCE_PROFILE_1_SLICE_REV2; GPU 1 exposes it via GPU_INSTANCE_PROFILE_1_SLICE. Then SetMigConfig(gpu=1, config={"1g.10gb": 1}):

  1. ParseMigProfile("1g.10gb") returns the GPU 0 variant (REV2), because GPU 0 was walked first.
  2. On GPU 1, GetGpuInstanceProfileInfo(REV2) returns ERROR_NOT_SUPPORTED.
  3. SetMigConfig fails and leaves GPU 1 cleared.

Reproduction / workaround

Reproduced and worked around in mig-parted PR #372, which adds a per-device resolver plus a test (TestSetMigConfigResolvesMigProfileOnTargetGPU in pkg/mig/config/config_test.go) that mocks the heterogeneous case. Originating report: mig-parted#157.

Consumer-side workaround: resolve profile names against the per-device Device.GetMigProfiles() rather than the devicelib cache.

Possible directions

  • A. Drop the dedup in (*devicelib).VisitMigProfiles; return one entry per (GPU, name). Changes the observable shape of GetMigProfiles/ParseMigProfile.
  • B. Keep the dedup, but have ParseMigProfile return a typed ErrAmbiguous when devices disagree on GI/CI IDs for a name.
  • C. Document the current behavior as "device-agnostic, first-walked wins" and steer NVML-bound callers to the per-device Device.GetMigProfiles().

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with VisitMigProfiles and GetMigProfiles in pkg/nvlib/device/device.go, then read ParseMigProfile in pkg/nvlib/device/mig_profile.go and compare the per-device traversal. Reproduce the heterogeneous-profile case described in the issue and inspect TestSetMigConfigResolvesMigProfileOnTargetGPU for expected consumer behavior. The issue is complete only after a direction among A–C is selected and covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.