installing gitops fails occasionally with connection refused on eksa webhook
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
I’ve seen the CloudStackLegacyFlux e2e test fail a number of times now (randomly) with the error
```
2022-08-26T03:50:36.132Z V0 ❌ Error when installing GitOps toolkits on workload cluster; EKS-A will continue with cluster creation, but GitOps will not be enabled {"error": "installing GitHub gitops: executing flux bootstrap github: ► connecting to github.c
om\n► cloning branch \"main\" from Git repository \"https://github.com/that-jetpack-guy/spacecraft-aws-eks-anywhere-test-3be30196-e0c1-457e-93d7-98de27788052-129.git\"\n✔ cloned repository\n► generating component manifests\n✔ generated component manifests\n✔ component m
anifests are up to date\n► installing components in \"default\" namespace\n✔ installed components\n✔ reconciled components\n► determining if source secret \"default/flux-system\" exists\n✔ source secret up to date\n► generating sync manifests\n✔ generated sync manifests
\n✔ sync manifests are up to date\n► applying sync manifests\n✔ reconciled sync configuration\n◎ waiting for Kustomization \"default/default\" to be reconciled\n✗ CloudStackDatacenterConfig/default/main-i-0ff4e-5b8dff2 apply failed, error: Internal error occurred: faile
d calling webhook \"validation.cloudstackdatacenterconfig.anywhere.amazonaws.com\": Post \"[https://eksa-webhook-service.eksa-system.svc:443/validate-anywhere-eks-amazonaws-com-v1alpha1-cloudstackdatacenterconfig?timeout=10s](https://eksa-webhook-service.eksa-system.svc/validate-anywhere-eks-amazonaws-com-v1alpha1-cloudstackdatacenterconfig?timeout=10s)\": dial tcp 10.109.213.213:443: connect: conne
ction refused\nCustomResourceDefinition/alerts.notification.toolkit.fluxcd.io configured\nCustomResourceDefinition/buckets.source.toolkit.fluxcd.io configured\nCustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io configured\nCustomResourceDefinition/helmcha
rts.source.toolkit.fluxcd.io configured\nCustomResourceDefinition/helmreleases.helm.toolkit.fluxcd.io configured\nCustomResourceDefinition/helmrepositories.source.toolkit.fluxcd.io configured\nCustomResourceDefinition/kustomizations.kustomize.toolkit.fluxcd.io configure
d\nCustomResourceDefinition/providers.notification.toolkit.fluxcd.io configured\nCustomResourceDefinition/receivers.notification.toolkit.fluxcd.io configured\nNamespace/default configured\n\n► confirming components are healthy\n✔ helm-controller: deployment ready\n✔ kus
tomize-controller: deployment ready\n✔ notification-controller: deployment ready\n✔ source-controller: deployment ready\n✔ all components are healthy\n✗ bootstrap failed with 1 health check failure(s)\n"}
```
which indicates to me that we are installing gitops before some eksa-webhooks are available. We should wait to install gitops until after the eksa pod is ready so that the webhooks are available.
**What you expected to happen**:
I would expect this operation to succeed without a connection refused error
**How to reproduce it (as minimally and precisely as possible)**:
Run the `TestCloudStackUpgradeMulticlusterWorkloadClusterWithFluxLegacy` a number of times
**Anything else we need to know?**:
**Environment**:
- EKS Anywhere Release: main
- EKS Distro Release:
Contributor guide
Research direction
Run TestCloudStackUpgradeMulticlusterWorkloadClusterWithFluxLegacy repeatedly and inspect the EKS Anywhere webhook readiness and GitOps installation sequence around the reported connection-refused error. Done means the test no longer intermittently fails because the eksa webhook is unavailable when GitOps is installed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- infrastructure, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100