aws-samples / aws-samples/sample-eks-enterprise-quickstart
eks-gpu-stack: gpu_operator_mofed_enabled sets non-existent Helm value "mofedDriver.enabled" (dead variable); real key is driver.rdma.*
- Dominant language
- Shell
- Stars
- 5
- Forks
- 3
- Avg merge
- 34m
- Merged PRs (30d)
- 2
Description
## Background
While fact-checking the companion blog post ("EKS 上的 GPU 工作负载:节点、网络与高性能存储的架构实践", §8.1) against this repo's source, found that the Operator-mode `gpu-operator` `helm_release` in `terraform/modules/eks-gpu-stack/main.tf` sets a Helm value key that doesn't exist in the actual NVIDIA GPU Operator chart.
## The bug
`terraform/modules/eks-gpu-stack/main.tf:397-399`:
```hcl
mofedDriver = {
enabled = var.gpu_operator_mofed_enabled
}
```
This is a **top-level** `mofedDriver` key, driven by `var.gpu_operator_mofed_enabled` (`terraform/variables.tf:545`, default `false`, description: *"Operator's MOFED driver. Default false because AWS EFA plugin owns /dev/infiniband/uverbs*."*). The intent is documented in the comment right above the `helm_release`:
```
# mofedDriver.enabled=false / driver.rdma.enabled=false — AWS EFA plugin
# owns /dev/infiniband/uverbs*
```
## Verified against upstream
Checked `NVIDIA/gpu-operator` at the **exact pinned tag** this repo uses (`gpu_operator_version = "v25.3.4"`):
- `deployments/gpu-operator/values.yaml` — no top-level `mofedDriver` key anywhere. The real field lives nested under `driver.rdma`:
```yaml
driver:
rdma:
enabled: false
useHostMofed: false
```
- `api/nvidia/v1/clusterpolicy_types.go` — `UseHostMOFED *bool \`json:"useHostMofed,omitempty"\`` is a field on `GPUDirectRDMASpec`, itself nested under the driver spec's `RDMA` field. No `MofedDriver` type exists anywhere in the CRD.
- `gh api search/code -f q='mofedDriver repo:NVIDIA/gpu-operator'` → 0 hits.
- The chart ships **no `values.schema.json`**, so Helm doesn't reject unknown values — `helm upgrade --install` / `terraform apply` succeeds silently with the dead key present. Nothing in `terraform plan`, CI, or a normal deploy would surface this.
## Practical impact
Currently **latent, not active**: `driver.rdma.enabled` is hardcoded to `false` a few lines above (`main.tf:390-392`) regardless of the variable, so the *default* outcome (MOFED/RDMA off, AWS EFA plugin owns `/dev/infiniband/uverbs*`) still happens to be correct today.
But `var.gpu_operator_mofed_enabled` itself is **fully dead** — flipping it to `true` has zero effect on the rendered chart, and there is currently no working path in this module to enable host-MOFED-based GPUDirect RDMA through Operator mode.
## Suggested fix
Fold the dead top-level block into the real `driver.rdma` field instead of keeping it separate from the hardcoded one:
```hcl
driver = {
enabled = var.gpu_operator_driver_enabled
rdma = {
enabled = var.gpu_operator_mofed_enabled
useHostMofed = var.gpu_operator_mofed_enabled
}
}
```
Drop the top-level `mofedDriver` block and the now-redundant hardcoded `rdma.enabled = false`, and update the comment on line 370 to drop the `mofedDriver.enabled=false` half.
## Where this was caught
Cross-checking `eks-blog/docs/2. ... GPU篇.docx` §8.1 against this repo: the blog post's claim *"GPU Operator has no `mofedDriver.enabled`, use `devicePlugin.env` `MOFED_ENABLED` instead"* is correct for the **standalone `nvidia-device-plugin` chart** used in Standard mode (§8.2), but doesn't apply to Operator mode, which uses `driver.rdma.*` instead — this module's Operator-mode code currently uses neither correctly.
Repo state checked: commit `d4d926923b727b702ad5175d51bcd32b81e716fb`.
Contributor guide
Research direction
Start in terraform/modules/eks-gpu-stack/main.tf at lines 370 and 390-399, then check terraform/variables.tf:545 to understand the variable's intended behavior. Inspect the Operator-mode gpu-operator helm_release values and validate the rendered configuration or terraform plan. Done means the dead top-level mofedDriver block is gone, driver.rdma uses the variable, and the comment matches the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, terraform
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100