GPU Module kick off

Title

Introduce Kyma GPU Module wrapping NVIDIA GPU Operator for Gardener clusters

Customer Problem

Customers want to run GPU-based workloads on Kyma clusters, such as AI/ML inference, training jobs, and other compute-intensive workloads.

Today, even when the underlying Gardener worker nodes already support GPUs and the OS image includes the required driver support path, customers still do not get a simple, Kyma-native way to enable and manage the NVIDIA software stack needed in the cluster.

This creates several problems:

• GPU enablement is not exposed as a standard Kyma capability
• Customers would need to install and manage NVIDIA components manually
• Operational setup is inconsistent across clusters
• Day-2 operations such as configuration, upgrades, and lifecycle handling are not aligned with the Kyma module model
• Platform teams cannot offer a clear, supported GPU experience as part of Kyma
• Garden upgrades could introduce inconsistencies

As a result, GPU workloads are harder to adopt, harder to operate, and less predictable for both customers and platform owners.

Goal

Provide a Kyma GPU Module that wraps the NVIDIA GPU Operator and offers GPU support in a Kyma-native, supported, and lifecycle-managed way.

The module should assume that:

• the underlying Gardener infrastructure supports GPU worker nodes
• the worker OS is Garden Linux
• required GPU driver support is already available on the node image / OS layer

The Kyma module is responsible for enabling and managing the Kubernetes-level NVIDIA stack, not for provisioning GPU infrastructure or maintaining OS drivers.

Scope

In scope

• Introduce a new Kyma GPU Module
• Wrap and deploy NVIDIA GPU Operator through the Kyma module model
• Define installation and configuration flow for GPU support in a Kyma cluster
• Ensure compatibility with Gardener clusters using Garden Linux-based worker nodes with driver support already present
• Provide a supported way for customers to consume GPU resources in workloads
• Define basic lifecycle handling:
  • installation
  • reconciliation
  • upgrade strategy
  • removal / cleanup behavior
• Document prerequisites, limitations, and supported setup

Out of scope

• Provisioning GPU node pools / machine types in hyperscalers
• Managing GPU drivers at OS image level
• Custom support for unsupported operating systems
• Full multi-vendor accelerator support beyond NVIDIA
• Advanced scheduling abstractions beyond standard Kubernetes GPU resource consumption

Proposed Solution

Create a Kyma module that packages and manages the NVIDIA GPU Operator.

The module would:

• install the operator into the cluster
• expose Kyma-managed configuration
• rely on Gardener + Garden Linux prerequisites being fulfilled underneath
• provide a standard Kyma entry point for enabling GPU support
• align GPU capabilities with Kyma module lifecycle and support model

This keeps the responsibility split clear:

• Gardener / worker OS layer → GPU-capable nodes and driver prerequisites
• Kyma GPU Module → NVIDIA operator and in-cluster GPU enablement
• Customer workloads → actual consumption of GPU resources

What needs to be done

• Define the Kyma GPU Module scope and ownership boundaries
• Validate assumptions around Garden Linux and preinstalled / supported NVIDIA driver path
• Decide how the module is enabled:
  • default optional module
  • per-cluster activation
  • configuration via custom resource
• Package NVIDIA GPU Operator as part of the module
• Define module configuration surface:
  • enablement
  • versioning
  • optional operator settings
• Define reconciliation and upgrade behavior
• Define failure handling and status reporting
• Verify compatibility with Gardener worker pools that expose GPUs
• Test workload consumption of GPU resources from user namespaces
• Document prerequisites for customers and operators
• Document unsupported scenarios and known limitations
• Define release/support expectations for the module

Acceptance Criteria

• A Kyma GPU Module exists and can be enabled on supported clusters
• The module installs and manages NVIDIA GPU Operator successfully
• GPU-capable nodes with Garden Linux and required driver support are detected and usable
• Customer workloads can request and consume GPU resources through standard Kubernetes mechanisms
• Module lifecycle is handled through Kyma conventions
• Prerequisites and limitations are documented

Open Questions

• How much configuration of NVIDIA GPU Operator should be exposed through the Kyma API?
• Do we support only clusters where drivers are already present, or do we also want future support for driver management paths?
• Is this module enabled only for specific worker pools / cluster profiles?
• What is the minimum supported Gardener / Kubernetes / Garden Linux combination?
• Do we want a minimal first version, or also include observability / monitoring for GPU resources?

Steps:

• Install an NVIDIA Operator manually, be familiar with the configuration options (https://github.com/gardenlinux/gardenlinux-nvidia-installer?tab=readme-ov-file#deploying-nvidia-gpu-operator-with-helm)
• Check where Garden is with China (Sovereign cloud)
• Check if Shoot could configure two different OS for two different Nodes (If the GPU Nodes are always on Garden Linux)?
• Check if we need multiple NVIDIA operator installers on the same cluster (if above is true)
• Make an ACR document for the Module and propose an architectural design

Start with the NVIDIA GPU Operator deployment guidance linked in the checklist and the Kyma module model. Review the open questions and prepare the ACR document with an architectural design covering enablement, configuration, lifecycle, prerequisites, and ownership boundaries. Done means the module scope and supported setup are defined against the listed acceptance criteria.