Custom CA for vCenter Works as Expected until Logout/Cleanup
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**: EKS Anywhere is successfully respecting the certificate settings until the logout step, which is causing docker containers to remain/not be cleaned up
```log
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
674c720feb8a public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 13 minutes ago Up 13 minutes eksa_1773691080748918511
c0cc09fbeeed public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 49 minutes ago Up 49 minutes eksa_1773688943312547476
ac8c7bb9c8c3 public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 2 hours ago Up 2 hours eksa_1773683755717527403
688fa20fc639 public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 13 days ago Up 13 days eksa_1772490377109977457
42ab2a6a4193 public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 13 days ago Up 13 days eksa_1772490279510500287
7f73c3d99864 public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 13 days ago Up 13 days eksa_1772490154947009845
d668de19acc5 public.ecr.aws:443/eks-anywhere/cli-tools:v0.24.4-eks-a-115 "sleep infinity" 13 days ago Up 13 days eksa_1772488659839382920
```
**What you expected to happen**: Successful logout
**How to reproduce it (as minimally and precisely as possible)**: Custom CA for vSphere, perform cluster upgrade as normal `eksctl anywhere upgrade cluster -f clusterconfig.yml`
**Anything else we need to know?**:
```yaml
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: VSphereDatacenterConfig
metadata:
name: dev
spec:
datacenter: # REDACTED
insecure: false
network: # REDACTED
server: 172.30.1.10
thumbprint: # REDACTED
```
```log
2026-03-16T20:07:21.187Z V0 🎉 Cluster upgraded!
2026-03-16T20:07:21.187Z V4 Task finished {"task_name": "write-cluster-config", "duration": "4.311002ms"}
2026-03-16T20:07:21.187Z V4 ----------------------------------
2026-03-16T20:07:21.187Z V4 Task start {"task_name": "post-cluster-upgrade"}
2026-03-16T20:07:21.187Z V3 Resuming all workload clusters after management cluster upgrade
2026-03-16T20:07:21.187Z V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1773691080748918511 kubectl get clusters.cluster.x-k8s.io -o json --kubeconfig dev/dev-eks-a-cluster.kubeconfig --namespace eksa-system"}
2026-03-16T20:07:21.400Z V0 Cleaning up backup resources
2026-03-16T20:07:21.402Z V4 Task finished {"task_name": "post-cluster-upgrade", "duration": "214.80003ms"}
2026-03-16T20:07:21.402Z V4 ----------------------------------
2026-03-16T20:07:21.402Z V4 Task start {"task_name": "upgrade-curated-packages"}
2026-03-16T20:07:21.402Z V4 Task finished {"task_name": "upgrade-curated-packages", "duration": "5.52µs"}
2026-03-16T20:07:21.402Z V4 ----------------------------------
2026-03-16T20:07:21.402Z V4 Tasks completed {"duration": "9m19.206904004s"}
2026-03-16T20:07:21.402Z V3 Logging out from current govc session
2026-03-16T20:07:21.402Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=172.30.1.10 -e GOVC_INSECURE=false -e GOVC_DATACENTER=Datacenter1 -e GOVC_TLS_KNOWN_HOSTS=dev/generated/govc_known_hosts eksa_1773691080748918511 govc session.logout"}
2026-03-16T20:07:21.517Z V9 docker {"stderr": "govc: Post \"https://172.30.1.10/sdk\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n"}
2026-03-16T20:07:21.517Z V-2 Closer failed {"closerType": "*dependencies.Dependencies", "error": "govc returned error when logging out: govc: Post \"https://172.30.1.10/sdk\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n"}
```
**Environment**:
- EKS Anywhere Release: v0.24.4
- EKS Distro Release: 1.34
Contributor guide
Research direction
Start by tracing the logout cleanup path around the *dependencies.Dependencies closer and the `govc session.logout` command shown in the logs. Check how the custom CA and `GOVC_TLS_KNOWN_HOSTS` settings reach that command, then reproduce the upgrade and verify logout succeeds without leaving Docker containers running.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100