aws-samples / aws-samples/sample-eks-enterprise-quickstart

eks-gpu-stack: gpu_operator_mofed_enabled sets non-existent Helm value "mofedDriver.enabled" (dead variable); real key is driver.rdma.*

Open Beginner friendly
#13 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
5
Forks
3
Avg merge
34m
Merged PRs (30d)
2

Description

## Background

While fact-checking the companion blog post ("EKS 上的 GPU 工作负载:节点、网络与高性能存储的架构实践", §8.1) against this repo's source, found that the Operator-mode `gpu-operator` `helm_release` in `terraform/modules/eks-gpu-stack/main.tf` sets a Helm value key that doesn't exist in the actual NVIDIA GPU Operator chart.

## The bug

`terraform/modules/eks-gpu-stack/main.tf:397-399`:

```hcl
mofedDriver = {
enabled = var.gpu_operator_mofed_enabled
}
```

This is a **top-level** `mofedDriver` key, driven by `var.gpu_operator_mofed_enabled` (`terraform/variables.tf:545`, default `false`, description: *"Operator's MOFED driver. Default false because AWS EFA plugin owns /dev/infiniband/uverbs*."*). The intent is documented in the comment right above the `helm_release`:

```
# mofedDriver.enabled=false / driver.rdma.enabled=false — AWS EFA plugin
# owns /dev/infiniband/uverbs*
```

## Verified against upstream

Checked `NVIDIA/gpu-operator` at the **exact pinned tag** this repo uses (`gpu_operator_version = "v25.3.4"`):

- `deployments/gpu-operator/values.yaml` — no top-level `mofedDriver` key anywhere. The real field lives nested under `driver.rdma`:
```yaml
driver:
rdma:
enabled: false
useHostMofed: false
```
- `api/nvidia/v1/clusterpolicy_types.go` — `UseHostMOFED *bool \`json:"useHostMofed,omitempty"\`` is a field on `GPUDirectRDMASpec`, itself nested under the driver spec's `RDMA` field. No `MofedDriver` type exists anywhere in the CRD.
- `gh api search/code -f q='mofedDriver repo:NVIDIA/gpu-operator'` → 0 hits.
- The chart ships **no `values.schema.json`**, so Helm doesn't reject unknown values — `helm upgrade --install` / `terraform apply` succeeds silently with the dead key present. Nothing in `terraform plan`, CI, or a normal deploy would surface this.

## Practical impact

Currently **latent, not active**: `driver.rdma.enabled` is hardcoded to `false` a few lines above (`main.tf:390-392`) regardless of the variable, so the *default* outcome (MOFED/RDMA off, AWS EFA plugin owns `/dev/infiniband/uverbs*`) still happens to be correct today.

But `var.gpu_operator_mofed_enabled` itself is **fully dead** — flipping it to `true` has zero effect on the rendered chart, and there is currently no working path in this module to enable host-MOFED-based GPUDirect RDMA through Operator mode.

## Suggested fix

Fold the dead top-level block into the real `driver.rdma` field instead of keeping it separate from the hardcoded one:

```hcl
driver = {
enabled = var.gpu_operator_driver_enabled
rdma = {
enabled = var.gpu_operator_mofed_enabled
useHostMofed = var.gpu_operator_mofed_enabled
}
}
```

Drop the top-level `mofedDriver` block and the now-redundant hardcoded `rdma.enabled = false`, and update the comment on line 370 to drop the `mofedDriver.enabled=false` half.

## Where this was caught

Cross-checking `eks-blog/docs/2. ... GPU篇.docx` §8.1 against this repo: the blog post's claim *"GPU Operator has no `mofedDriver.enabled`, use `devicePlugin.env` `MOFED_ENABLED` instead"* is correct for the **standalone `nvidia-device-plugin` chart** used in Standard mode (§8.2), but doesn't apply to Operator mode, which uses `driver.rdma.*` instead — this module's Operator-mode code currently uses neither correctly.

Repo state checked: commit `d4d926923b727b702ad5175d51bcd32b81e716fb`.

Contributor guide

Open the contributing guide

Research direction

Start in terraform/modules/eks-gpu-stack/main.tf at lines 370 and 390-399, then check terraform/variables.tf:545 to understand the variable's intended behavior. Inspect the Operator-mode gpu-operator helm_release values and validate the rendered configuration or terraform plan. Done means the dead top-level mofedDriver block is gone, driver.rdma uses the variable, and the comment matches the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, terraform
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.