[EKS] [BUG]: eks:certificates-controller leaves CSR in pending state if two nodes with same IP/HostName join and leave back to back
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
- Please help to investigate the described issue
- Please consider open-sourcing eks-certificates-controller as it's an important part of EKS control plane and there's little documentation around it.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Our organization has a fairly busy EKS cluster that runs Gitlab jobs. We use Karpenter to provision spot instances as Gitlab jobs are scheduled on the cluster as Pods. When the Pod finishes and the node is empty, Karpenter will delete it immediately.
We noticed transient, random failures when new nodes join the cluster, which is associated with the symptoms that the `kubelet-serving` CSR for the failing node was stuck on `Pending` state forever, and `kubectl logs` and `kubectl exec` fails with `error dialing backend: remote error: tls: internal error.` as expected when node CSR isn't issued.
Interestingly, we could correlate these failures to two worker nodes with the same IP/HostName joining and leaving the cluster back to back. __It looks like eks' certificates-controller is skipping CSRs if a different instance with the same HostName recently left the cluster.__
Without access to eks certificate-controller source code, we cannot be sure about our theory, but here are some strong signals that point to this direction.
### APIServer timeline
1. 2023-02-01T09:56:31.522Z - EC2 instance __A__ with `ip-10-14-72-14.us-west-2.compute.internal` as hostname joins cluster, and created CSR `csr-tdzwj` to APIServer.
2. 2023-02-01T09:56:47.386Z - `eks:certificate-controller` kicks in, did SubjectAccessReview, approved CSR `csr-tdzwj` and updated its status.
3. 2023-02-01T09:59:15.485Z - EC2 instance __A__ leaves the cluster. (last k8s api audit log from it)
4. 2023-02-01T10:00:24.529Z - EC2 instance __B__ with `ip-10-14-72-14.us-west-2.compute.internal` joins cluster, and created CSR `csr-rqzgg` for its `kubelet-signing` token.
5. - the CSR `csr-rqzgg` never got approved nor issued.
### Cloudtrail logs
At AWS side, I see an `ec2:DescribeInstances` call around the same time as eks:certificate-controller approves the first CSR for `ip-10-14-72-14.us-west-2.compute.internal`:
```json
{
"eventTime": "2023-02-01T09:56:47Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "DescribeInstances",
"awsRegion": "us-west-2",
"sourceIPAddress": "eks.amazonaws.com",
"userAgent": "eks.amazonaws.com",
"requestParameters": {
"maxResults": 190,
"instancesSet": {},
"filterSet": {
"items": [
{
"name": "vpc-id",
"valueSet": {
"items": [
{
"value": ""
}
]
}
},
{
"name": "private-dns-name",
"valueSet": {
"items": [
{
"value": "ip-10-14-72-14.us-west-2.compute.internal"
},
{
"value": "ip-10-14-72-12.us-west-2.compute.internal"
},
{
"value": "ip-10-14-72-29.us-west-2.compute.internal"
}
]
}
}
]
}
},
"requestID": "1d83813c-cc20-4ff5-8e31-ca6704932cbb"
}
```
(only cherry-picked relevant fields)
Looks like eks:certificates-controller verifies EC2 existence by HostName + VPCId during CSR validation in a batch. __However I couldn't find any similar call for the second (skipped) CSR__. It looks like eks:certificates-controller just completely ignored the CSR for some reason.
**Are you currently working around this issue?**
This issue is annoying and hard to work around, as Karpenter just launches EC2s randomly among subnets, and EC2 reuses and assigns IPs randomly. It's hard to mitigate this issue.
**Additional context**
- Version information
- EKS: `1.24-eks.3`
- Worker nodes: we are using the default AmazonLinux AMI without customization. at the time of this issue it's `amazon-eks-node-1.24-v20230105`. Note that we run both AMD64 and ARM64 workloads, and the issue occur to both of them.
- Karpenter: `v0.22.1`
- clusterArn, account information and raw logs are available if needed.
Thank you very much.
Contributor guide
Research direction
No repository file, test, or entry point is identified; the report concerns the proprietary EKS certificates-controller and requests investigation or open-sourcing. Start by reviewing the CSR timeline, CloudTrail evidence, and Kubernetes CSR behavior described in the issue. Done would require a confirmed cause and an actionable fix or documented resolution from the EKS maintainers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure, security
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100