aws / aws/containers-roadmap

[EKS] [bug]: eks api unable to call webhook's when using the custom cni networking

Open
#1,215 4 comments 16 reactions 0 assignees View on GitHub
EKS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
eks api unable to call webhook's when using the custom cni networking

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I am trying to create clusterissuer object of cert-manager i.e https://cert-manager.io/docs/

sboga@C02SW14YG8WL ~ % kubectl api-resources|grep -i clusterissuers
clusterissuers cert-manager.io false ClusterIssuer

And creating this object involves api contacting the webhook api svc for validation which is failing

Logs from kubectl apply -f /tmp/file -v=9

I0106 20:24:00.361568 63216 request.go:1097] Request Body: {"apiVersion":"cert-manager.io/v1","kind":"ClusterIssuer","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"cert-manager.io/v1\",\"kind\":\"ClusterIssuer\",\"metadata\":{\"annotations\":{},\"name\":\"letsencrypt-prod1\"},\"spec\":{\"acme\":{\"email\":\"hostmaster@roku.com\",\"privateKeySecretRef\":{\"name\":\"letsencrypt-prod\"},\"server\":\"https://acme-v02.api.letsencrypt.org/directory\",\"solvers\":[{\"dns01\":{\"route53\":{\"hostedZoneID\":\"Z27VCBU17XD4X4\",\"region\":\"us-east-1\",\"role\":\"arn:aws:iam::088414020449:role/dns-rokulabs-net-manager-role\"}},\"selector\":{\"dnsZones\":[\"rokulabs.net\"]}},{\"dns01\":{\"route53\":{\"hostedZoneID\":\"Z3S2XVQIXW67TY\",\"region\":\"us-east-1\",\"role\":\"arn:aws:iam::824163854805:role/dns-msc-dev-roku-com-manager-role\"}},\"selector\":{\"dnsZones\":[\"msc-dev.roku.com\"]}}]}}}\n" },"name":"letsencrypt-prod1"},"spec":{"acme":{"email":"hostmaster@roku.com","privateKeySecretRef":{"name":"letsencrypt-prod"},"server":"https://acme-v02.api.letsencrypt.org/directory","solvers":[{"dns01":{"route53":{"hostedZoneID":"Z27VCBU17XD4X4","region":"us-east-1","role":"arn:aws:iam::088414020449:role/dns-rokulabs-net-manager-role"}},"selector":{"dnsZones":["rokulabs.net"]}},{"dns01":{"route53":{"hostedZoneID":"Z3S2XVQIXW67TY","region":"us-east-1","role":"arn:aws:iam::824163854805:role/dns-msc-dev-roku-com-manager-role"}},"selector":{"dnsZones":["msc-dev.roku.com "]}}]}}}
I0106 20:24:00.361765 63216 round_trippers.go:423] curl -k -v -XPOST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.19.0 (darwin/amd64) kubernetes/e199641" 'https://3CAC058C6DE86A596295535EC5B4E9C7.gr7.us-west-2.eks.amazonaws.com/apis/cert-manager.io/v1/clusterissuers?fieldManager=kubectl-client-side-apply '
I0106 20:24:00.408237 63216 round_trippers.go:443] POST https://3CAC058C6DE86A596295535EC5B4E9C7.gr7.us-west-2.eks.amazonaws.com/apis/cert-manager.io/v1/clusterissuers?fieldManager=kubectl-client-side-apply 500 Internal Server Error in 46 milliseconds
I0106 20:24:00.408276 63216 round_trippers.go:449] Response Headers:
I0106 20:24:00.408283 63216 round_trippers.go:452] Content-Type: application/json
I0106 20:24:00.408290 63216 round_trippers.go:452] Content-Length: 479
I0106 20:24:00.408296 63216 round_trippers.go:452] Date: Thu, 07 Jan 2021 04:24:00 GMT
I0106 20:24:00.408302 63216 round_trippers.go:452] Audit-Id: ec534821-9753-4377-8f7b-7b8ab014243e
I0106 20:24:00.408308 63216 round_trippers.go:452] Cache-Control: no-cache, private
I0106 20:24:00.408355 63216 request.go:1097] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: Address is not allowed","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: Address is not allowed"}]},"code":500}
I0106 20:24:00.408706 63216 helpers.go:216] server response object: [{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: Address is not allowed",
"reason": "InternalError",
"details": {
"causes": [
{
"message": "failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: Address is not allowed"
}
]
},
"code": 500
}]
F0106 20:24:00.408756 63216 helpers.go:115] Error from server (InternalError): Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: Address is not allowed
Region: us-west-2
Cluster Name:
Instance ID(s):

**Are you currently working around this issue?**
Did not find any

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the kubectl apply request for a cert-manager ClusterIssuer in an EKS cluster using custom CNI networking, then inspect the cert-manager webhook call and the reported "Address is not allowed" response. Done means identifying the networking failure and restoring successful EKS API calls to the webhook.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.