aws / aws/containers-roadmap

[EKS] [eks-node-monitoring-agent]: DCGM Error: Message :"failed to initialize DCGM: Error connecting to nv-hostengine: Host engine connection invalid/disconnected"

Open
#2,555 12 comments 19 reactions 0 assignees View on GitHub
EKS Nodes
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

Issue Overview:
eks-node-monitoring-agent can not connect to dcgm and it transitions the node to AcceleratedHardwareReady = False. DCGM is installed and runs fine with GPU-Operator (latest version)

Findings:
- no

Impact:
• Node not healthy and EKS auto repair replaces the node

Expected Behavior:
• monitoring agent should be able to connect to dcgm and not set AcceleratedHardwareReady = False

Requesting Labels: EKS, Amazon Elastic Kubernetes Service

```
{"level":"error","ts":"2025-03-04T07:57:23Z","msg":"failed to reconcile DCGM state","hostname":"ip-10-10-15-1.eu-west-1.compute.internal","monitor":"nvidia","error":"failed to initialize DCGM: Error connecting to nv-hostengine: Host engine connection invalid/disconnected","stacktrace":"golang.a2z.com/EKSNodeMonitoringAgent/internal/monitor/nvidia.(*NvidiaMonitor).reconcileDcgm\n\t/local/p4clients/pkgbuild-const/workspace/src/EKSNodeMonitoringAgent/internal/monitor/nvidia/monitor.go:62\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:259\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:227\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:204\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:259\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/local/p4clients/pkgbuild-const/workspace/tmp/gomodcache/k8s.io/apimachinery@v0.31.1/pkg/util/wait/backoff.go:170"}
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the DCGM connection failure in the EKS node monitoring agent and inspect internal/monitor/nvidia/monitor.go at the reconcileDcgm call shown in the trace. Determine why nv-hostengine is reported as disconnected despite DCGM working with GPU-Operator. Done means the agent connects to DCGM and no longer sets AcceleratedHardwareReady to False for this condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
cloud, infrastructure, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.