aws / aws/eks-anywhere

Custom CA for vCenter Works as Expected until Logout/Cleanup

Open
#10,685 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**: EKS Anywhere is successfully respecting the certificate settings until the logout step, which is causing docker containers to remain/not be cleaned up
```log
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
674c720feb8a public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 13 minutes ago Up 13 minutes eksa_1773691080748918511
c0cc09fbeeed public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 49 minutes ago Up 49 minutes eksa_1773688943312547476
ac8c7bb9c8c3 public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 2 hours ago Up 2 hours eksa_1773683755717527403
688fa20fc639 public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 13 days ago Up 13 days eksa_1772490377109977457
42ab2a6a4193 public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 13 days ago Up 13 days eksa_1772490279510500287
7f73c3d99864 public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 13 days ago Up 13 days eksa_1772490154947009845
d668de19acc5 public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 13 days ago Up 13 days eksa_1772488659839382920
```

**What you expected to happen**: Successful logout

**How to reproduce it (as minimally and precisely as possible)**: Custom CA for vSphere, perform cluster upgrade as normal `eksctl anywhere upgrade cluster -f clusterconfig.yml`

**Anything else we need to know?**:
```yaml
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: VSphereDatacenterConfig
metadata:
name: dev
spec:
datacenter: # REDACTED
insecure: false
network: # REDACTED
server: 172.30.1.10
thumbprint: # REDACTED
```

```log
2026-03-16T20:07:21.187Z V0 🎉 Cluster upgraded!
2026-03-16T20:07:21.187Z V4 Task finished {"task_name": "write-cluster-config", "duration": "4.311002ms"}
2026-03-16T20:07:21.187Z V4 ----------------------------------
2026-03-16T20:07:21.187Z V4 Task start {"task_name": "post-cluster-upgrade"}
2026-03-16T20:07:21.187Z V3 Resuming all workload clusters after management cluster upgrade
2026-03-16T20:07:21.187Z V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1773691080748918511 kubectl get clusters.cluster.x-k8s.io -o json --kubeconfig dev/dev-eks-a-cluster.kubeconfig --namespace eksa-system"}
2026-03-16T20:07:21.400Z V0 Cleaning up backup resources
2026-03-16T20:07:21.402Z V4 Task finished {"task_name": "post-cluster-upgrade", "duration": "214.80003ms"}
2026-03-16T20:07:21.402Z V4 ----------------------------------
2026-03-16T20:07:21.402Z V4 Task start {"task_name": "upgrade-curated-packages"}
2026-03-16T20:07:21.402Z V4 Task finished {"task_name": "upgrade-curated-packages", "duration": "5.52µs"}
2026-03-16T20:07:21.402Z V4 ----------------------------------
2026-03-16T20:07:21.402Z V4 Tasks completed {"duration": "9m19.206904004s"}
2026-03-16T20:07:21.402Z V3 Logging out from current govc session
2026-03-16T20:07:21.402Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=172.30.1.10 -e GOVC_INSECURE=false -e GOVC_DATACENTER=Datacenter1 -e GOVC_TLS_KNOWN_HOSTS=dev/generated/govc_known_hosts eksa_1773691080748918511 govc session.logout"}
2026-03-16T20:07:21.517Z V9 docker {"stderr": "govc: Post \"https://172.30.1.10/sdk\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n"}
2026-03-16T20:07:21.517Z V-2 Closer failed {"closerType": "*dependencies.Dependencies", "error": "govc returned error when logging out: govc: Post \"https://172.30.1.10/sdk\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n"}
```

**Environment**:
- EKS Anywhere Release: v0.24.4
- EKS Distro Release: 1.34

Contributor guide

Open the contributing guide

Research direction

Start by tracing the logout cleanup path around the *dependencies.Dependencies closer and the `govc session.logout` command shown in the logs. Check how the custom CA and `GOVC_TLS_KNOWN_HOSTS` settings reach that command, then reproduce the upgrade and verify logout succeeds without leaving Docker containers running.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.