aws / aws/amazon-eks-pod-identity-webhook
Restarted pods request new CSR instead of using existing cert in k8s secret
- Dominant language
- Go
- Stars
- 691
- Forks
- 202
- Avg merge
- 4h 38m
- Merged PRs (30d)
- 1
Description
**What happened**:
Pods aren't annotated due to webhook service availability issue:
```
...
E0213 16:11:52.169236 1 certificate_manager.go:396] Certificate request was not signed: timed out waiting for the condition
E0213 16:27:20.169637 1 certificate_manager.go:396] Certificate request was not signed: timed out waiting for the condition
2020/02/13 16:30:29 http: TLS handshake error from 10.244.29.0:60094: no serving certificate available for the webhook, is the CSR approved?
```
This appears to happen when the pod restarts, causing mass CSR requests:
```
NAME AGE REQUESTOR CONDITION
csr-28lr7 3h8m system:serviceaccount:kube-system:aws-pod-id-production-aws-pod-identity-webhook Pending
csr-2fk56 4h53m system:serviceaccount:kube-system:aws-pod-id-production-aws-pod-identity-webhook Pending
csr-2jvt7 3h39m system:serviceaccount:kube-system:aws-pod-id-production-aws-pod-identity-webhook Pending
...
```
**What you expected to happen**:
Re-use of existing cert stored in kubernetes secret.
**How to reproduce it (as minimally and precisely as possible)**:
```
/webhook
--in-cluster
--namespace=kube-system
--service-name=aws-pod-id-production-aws-pod-identity-webhook
--tls-secret=pod-identity-webhook
--annotation-prefix=iam.amazonaws.com
--token-audience=sts.amazonaws.com
--logtostderr
```
**Environment**:
- AWS Region: `us-east-1`
- EKS Platform version (if using EKS, run `aws eks describe-cluster --name --query cluster.platformVersion`): `NA`
- Kubernetes version (if using EKS, run `aws eks describe-cluster --name --query cluster.version`): `1.15.6`
- Webhook Version: `v0.1.0`
Contributor guide
Research direction
Start at the /webhook entry point and inspect certificate_manager.go around the logged line 396, focusing on how the Kubernetes TLS secret and CSR flow are selected after a pod restart. Done means restarted pods reuse the existing pod-identity-webhook secret certificate and no longer create a mass of pending CSRs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- infrastructure, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100