[EKS] [request]: On create: only return ACTIVE when endpoint actually usable
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
When an EKS cluster is created the API reports an "ACTIVE" status before the endpoint can actually process requests. This means the first few attempts to use new clusters receive connection timeouts. All projects that create clusters have to implement retry logic for the first access to the api, usually when updating the aws-auth ConfigMap.
It would be super useful if the API reported an ACTIVE status on newly created clusters only once the endpoint was actually available to process requests. We've already waited over 10 minutes for the cluster to come up so waiting ~30 seconds more for it to actually be useable wouldn't be a big issue.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
The [Terraform EKS community module](https://github.com/terraform-aws-modules/terraform-aws-eks/issues/621) is trying to migrate from running kubectl in a shell to using the kubernetes provider for creating the aws-auth ConfigMap. This would help with cross-platform use. Unfortunately due to "ACTIVE" not meaning "USABLE" we've hit issues with chaining the two providers together.
The kubernetes provider itself [has refused](https://github.com/terraform-providers/terraform-provider-kubernetes/issues/96) to implement retry logic on connection timeouts.
**Are you currently working around this issue?**
Projects that create clusters have some form of retry loop with a sleep.
**Additional context**
Will potentially make other requests that deal with newly created clusters easier: #185, #254, #51
**Attachments**
- [Sample terraform code](https://gist.github.com/dpiddockcmp/23342f3b601b3432b1ea98ab61af6ba0)
- [eksctl retry logic](https://github.com/weaveworks/eksctl/blob/c98c5821e0fb0730d4f80ea22f9c5369d889f1ab/pkg/ctl/create/cluster.go#L351)
- [quickstart example retry logic](https://github.com/aws-quickstart/quickstart-amazon-eks/blob/29a1d8da363c8ffd8f3b7f1f8725d85d04657a3b/functions/source/KubeManifest/lambda_function.py#L37)
- [Terraform aws-eks module v7.0.1 retry logic](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/b69c8fb7596de59aa69fee020f8941ec6618f6b4/aws_auth.tf#L15..L22)
Contributor guide
Research direction
The payload names no repository files or tests for an implementation. Start by reviewing the EKS behavior described here and the linked retry paths in eksctl's cluster.go, the quickstart lambda_function.py, and Terraform's aws_auth.tf. Done means newly created clusters report ACTIVE only when their endpoint can process requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, terraform
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100