[Feature]: Merge custom MIG profiles with dynamically generated profiles
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.9k
- Forks
- 552
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
Is your feature request related to a specific problem?
GPU Operator currently supports either dynamically generated MIG profiles or a user-provided static MIG configuration.
When no custom ConfigMap is configured, MIG Manager generates hardware-aware profiles from the detected GPU capabilities. This keeps the profile set aligned with new GPU models, MIG profile changes, and upstream updates. The generated configuration also contains device-specific entries using device-filter.
However, when a custom MIG ConfigMap is configured, MIG Manager uses only that ConfigMap. The generated profiles are no longer available. Users therefore have to copy the complete upstream configuration, including all standard profiles and device-filter entries, and add their own profiles manually.
This creates an operational risk: the copied configuration will inevitably become stale as the upstream profile database evolves. Someone must periodically compare it with NVIDIA’s upstream 0400_configmap.yaml and update it, which is easy to miss across multiple clusters.
Describe the solution you would like
Please add an opt-in merge mode that combines the dynamically generated configuration with user-defined profiles.
For example:
migManager:
config:
name: custom-mig-parted-config
mergeWithGenerated: true
The resulting profile set would contain both the generated profiles and the custom profiles. A custom ConfigMap would only need to define organization-specific profiles, for example:
version: v1
mig-configs:
x100-80gb-10-20-20-20:
- device-filter: ["0x233010DE", "0x233110DE"]
devices: all
mig-enabled: true
mig-devices:
"1g.10gb": 1
"2g.20gb": 3
x100-80gb-20-20-40:
- device-filter: ["0x233010DE", "0x233110DE"]
devices: all
mig-enabled: true
mig-devices:
"2g.20gb": 2
"3g.40gb": 1
On an H100 80GB node, users would then be able to select both generated profiles such as all-1g.10gb, all-2g.20gb, and all-balanced, and custom profiles such as x100-80gb-20-20-40.
Describe alternatives you have considered
The current workaround is to vendor the entire upstream 0400_configmap.yaml into every deployment and append custom profiles to it. This works, but creates a permanent maintenance burden and a risk of silently falling behind upstream.
Another workaround would be to patch the generated ConfigMap using a Kubernetes Job or hook. This is racy because the generated ConfigMap is node-specific and may be recreated or overwritten by MIG Manager.
Additional details
Suggested merge semantics:
- Generated profiles remain available by default.
- Custom profiles are appended.
- If a custom profile has the same name as a generated profile, the custom definition takes precedence.
- Duplicate names should produce a clear log message.
device-filtermust work for both generated and custom profiles.- Invalid custom profiles should fail validation with an actionable error.
- Existing behavior must remain unchanged when merge mode is disabled.
This would preserve the convenience and freshness of dynamically generated profiles while allowing clusters to define a small set of stable, organization-specific combined layouts.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing MIG Manager's handling of the custom ConfigMap and dynamically generated profiles, using assets/state-mig-manager/0400_configmap.yaml as the reference configuration. Define how opt-in merging, custom-profile precedence, duplicate names, device-filter entries, and invalid-profile validation should work. Done means merge mode exposes both profile sets while disabled mode remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100