aws / aws/amazon-eks-pod-identity-webhook

Restarted pods request new CSR instead of using existing cert in k8s secret

Open
#35 8 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
691
Forks
202
Avg merge
4h 38m
Merged PRs (30d)
1

Description

**What happened**:

Pods aren't annotated due to webhook service availability issue:

```
...
E0213 16:11:52.169236 1 certificate_manager.go:396] Certificate request was not signed: timed out waiting for the condition
E0213 16:27:20.169637 1 certificate_manager.go:396] Certificate request was not signed: timed out waiting for the condition
2020/02/13 16:30:29 http: TLS handshake error from 10.244.29.0:60094: no serving certificate available for the webhook, is the CSR approved?
```

This appears to happen when the pod restarts, causing mass CSR requests:

```
NAME AGE REQUESTOR CONDITION
csr-28lr7 3h8m system:serviceaccount:kube-system:aws-pod-id-production-aws-pod-identity-webhook Pending
csr-2fk56 4h53m system:serviceaccount:kube-system:aws-pod-id-production-aws-pod-identity-webhook Pending
csr-2jvt7 3h39m system:serviceaccount:kube-system:aws-pod-id-production-aws-pod-identity-webhook Pending
...
```

**What you expected to happen**:

Re-use of existing cert stored in kubernetes secret.

**How to reproduce it (as minimally and precisely as possible)**:

```
/webhook
--in-cluster
--namespace=kube-system
--service-name=aws-pod-id-production-aws-pod-identity-webhook
--tls-secret=pod-identity-webhook
--annotation-prefix=iam.amazonaws.com
--token-audience=sts.amazonaws.com
--logtostderr
```

**Environment**:
- AWS Region: `us-east-1`
- EKS Platform version (if using EKS, run `aws eks describe-cluster --name --query cluster.platformVersion`): `NA`
- Kubernetes version (if using EKS, run `aws eks describe-cluster --name --query cluster.version`): `1.15.6`
- Webhook Version: `v0.1.0`

Contributor guide

Open the contributing guide

Research direction

Start at the /webhook entry point and inspect certificate_manager.go around the logged line 396, focusing on how the Kubernetes TLS secret and CSR flow are selected after a pod restart. Done means restarted pods reuse the existing pod-identity-webhook secret certificate and no longer create a mass of pending CSRs.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.