NVIDIA / NVIDIA/gpu-operator

[Feature]: Merge custom MIG profiles with dynamically generated profiles

Open
#2,913 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature lifecycle/frozen
Dominant language
Go
Stars
2.9k
Forks
552
Avg merge
2d 4h
Merged PRs (30d)
90

Description

Is your feature request related to a specific problem?

GPU Operator currently supports either dynamically generated MIG profiles or a user-provided static MIG configuration.

When no custom ConfigMap is configured, MIG Manager generates hardware-aware profiles from the detected GPU capabilities. This keeps the profile set aligned with new GPU models, MIG profile changes, and upstream updates. The generated configuration also contains device-specific entries using device-filter.

However, when a custom MIG ConfigMap is configured, MIG Manager uses only that ConfigMap. The generated profiles are no longer available. Users therefore have to copy the complete upstream configuration, including all standard profiles and device-filter entries, and add their own profiles manually.

This creates an operational risk: the copied configuration will inevitably become stale as the upstream profile database evolves. Someone must periodically compare it with NVIDIA’s upstream 0400_configmap.yaml and update it, which is easy to miss across multiple clusters.

Describe the solution you would like

Please add an opt-in merge mode that combines the dynamically generated configuration with user-defined profiles.

For example:

migManager:
  config:
    name: custom-mig-parted-config
    mergeWithGenerated: true

The resulting profile set would contain both the generated profiles and the custom profiles. A custom ConfigMap would only need to define organization-specific profiles, for example:

version: v1
mig-configs:
  x100-80gb-10-20-20-20:
    - device-filter: ["0x233010DE", "0x233110DE"]
      devices: all
      mig-enabled: true
      mig-devices:
        "1g.10gb": 1
        "2g.20gb": 3

  x100-80gb-20-20-40:
    - device-filter: ["0x233010DE", "0x233110DE"]
      devices: all
      mig-enabled: true
      mig-devices:
        "2g.20gb": 2
        "3g.40gb": 1

On an H100 80GB node, users would then be able to select both generated profiles such as all-1g.10gb, all-2g.20gb, and all-balanced, and custom profiles such as x100-80gb-20-20-40.

Describe alternatives you have considered

The current workaround is to vendor the entire upstream 0400_configmap.yaml into every deployment and append custom profiles to it. This works, but creates a permanent maintenance burden and a risk of silently falling behind upstream.

Another workaround would be to patch the generated ConfigMap using a Kubernetes Job or hook. This is racy because the generated ConfigMap is node-specific and may be recreated or overwritten by MIG Manager.

Additional details

Suggested merge semantics:

  • Generated profiles remain available by default.
  • Custom profiles are appended.
  • If a custom profile has the same name as a generated profile, the custom definition takes precedence.
  • Duplicate names should produce a clear log message.
  • device-filter must work for both generated and custom profiles.
  • Invalid custom profiles should fail validation with an actionable error.
  • Existing behavior must remain unchanged when merge mode is disabled.

This would preserve the convenience and freshness of dynamically generated profiles while allowing clusters to define a small set of stable, organization-specific combined layouts.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing MIG Manager's handling of the custom ConfigMap and dynamically generated profiles, using assets/state-mig-manager/0400_configmap.yaml as the reference configuration. Define how opt-in merging, custom-profile precedence, duplicate names, device-filter entries, and invalid-profile validation should work. Done means merge mode exposes both profile sets while disabled mode remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.