aws / aws/eks-anywhere

installing gitops fails occasionally with connection refused on eksa webhook

Open
#3,199 2 comments 0 reactions 0 assignees View on GitHub
area/gitops kind/bug team/cli
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
I’ve seen the CloudStackLegacyFlux e2e test fail a number of times now (randomly) with the error
```
2022-08-26T03:50:36.132Z V0 ❌ Error when installing GitOps toolkits on workload cluster; EKS-A will continue with cluster creation, but GitOps will not be enabled {"error": "installing GitHub gitops: executing flux bootstrap github: ► connecting to github.c
om\n► cloning branch \"main\" from Git repository \"https://github.com/that-jetpack-guy/spacecraft-aws-eks-anywhere-test-3be30196-e0c1-457e-93d7-98de27788052-129.git\"\n✔ cloned repository\n► generating component manifests\n✔ generated component manifests\n✔ component m
anifests are up to date\n► installing components in \"default\" namespace\n✔ installed components\n✔ reconciled components\n► determining if source secret \"default/flux-system\" exists\n✔ source secret up to date\n► generating sync manifests\n✔ generated sync manifests
\n✔ sync manifests are up to date\n► applying sync manifests\n✔ reconciled sync configuration\n◎ waiting for Kustomization \"default/default\" to be reconciled\n✗ CloudStackDatacenterConfig/default/main-i-0ff4e-5b8dff2 apply failed, error: Internal error occurred: faile
d calling webhook \"validation.cloudstackdatacenterconfig.anywhere.amazonaws.com\": Post \"[https://eksa-webhook-service.eksa-system.svc:443/validate-anywhere-eks-amazonaws-com-v1alpha1-cloudstackdatacenterconfig?timeout=10s](https://eksa-webhook-service.eksa-system.svc/validate-anywhere-eks-amazonaws-com-v1alpha1-cloudstackdatacenterconfig?timeout=10s)\": dial tcp 10.109.213.213:443: connect: conne
ction refused\nCustomResourceDefinition/alerts.notification.toolkit.fluxcd.io configured\nCustomResourceDefinition/buckets.source.toolkit.fluxcd.io configured\nCustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io configured\nCustomResourceDefinition/helmcha
rts.source.toolkit.fluxcd.io configured\nCustomResourceDefinition/helmreleases.helm.toolkit.fluxcd.io configured\nCustomResourceDefinition/helmrepositories.source.toolkit.fluxcd.io configured\nCustomResourceDefinition/kustomizations.kustomize.toolkit.fluxcd.io configure
d\nCustomResourceDefinition/providers.notification.toolkit.fluxcd.io configured\nCustomResourceDefinition/receivers.notification.toolkit.fluxcd.io configured\nNamespace/default configured\n\n► confirming components are healthy\n✔ helm-controller: deployment ready\n✔ kus
tomize-controller: deployment ready\n✔ notification-controller: deployment ready\n✔ source-controller: deployment ready\n✔ all components are healthy\n✗ bootstrap failed with 1 health check failure(s)\n"}
```

which indicates to me that we are installing gitops before some eksa-webhooks are available. We should wait to install gitops until after the eksa pod is ready so that the webhooks are available.

**What you expected to happen**:
I would expect this operation to succeed without a connection refused error

**How to reproduce it (as minimally and precisely as possible)**:
Run the `TestCloudStackUpgradeMulticlusterWorkloadClusterWithFluxLegacy` a number of times

**Anything else we need to know?**:

**Environment**:
- EKS Anywhere Release: main
- EKS Distro Release:

Contributor guide

Open the contributing guide

Research direction

Run TestCloudStackUpgradeMulticlusterWorkloadClusterWithFluxLegacy repeatedly and inspect the EKS Anywhere webhook readiness and GitOps installation sequence around the reported connection-refused error. Done means the test no longer intermittently fails because the eksa webhook is unavailable when GitOps is installed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
infrastructure, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.