[EKS] [eks-node-monitoring-agent]: DCGM Error: Message :"failed to initialize DCGM: Error connecting to nv-hostengine: Host engine connection invalid/disconnected"
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
Issue Overview:
eks-node-monitoring-agent can not connect to dcgm and it transitions the node to AcceleratedHardwareReady = False. DCGM is installed and runs fine with GPU-Operator (latest version)
Findings:
- no
Impact:
• Node not healthy and EKS auto repair replaces the node
Expected Behavior:
• monitoring agent should be able to connect to dcgm and not set AcceleratedHardwareReady = False
Requesting Labels: EKS, Amazon Elastic Kubernetes Service
```
{"level":"error","ts":"2025-03-04T07:57:23Z","msg":"failed to reconcile DCGM state","hostname":"ip-10-10-15-1.eu-west-1.compute.internal","monitor":"nvidia","error":"failed to initialize DCGM: Error connecting to nv-hostengine: Host engine connection invalid/disconnected","stacktrace":"golang.a2z.com/EKSNodeMonitoringAgent/internal/monitor/nvidia.(*NvidiaMonitor).reconcileDcgm\n\t/local/p4clients/pkgbuild-const/workspace/src/EKSNodeMonitoringAgent/internal/monitor/nvidia/monitor.go:62\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:259\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:259\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:170"}
```
Contributor guide
Research direction
Start by reproducing the DCGM connection failure in the EKS node monitoring agent and inspect internal/monitor/nvidia/monitor.go at the reconcileDcgm call shown in the trace. Determine why nv-hostengine is reported as disconnected despite DCGM working with GPU-Operator. Done means the agent connects to DCGM and no longer sets AcceleratedHardwareReady to False for this condition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- cloud, infrastructure, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100