aws / aws/amazon-eks-pod-identity-webhook
Prometheus scraper misconfigured in the k8s service
- Dominant language
- Go
- Stars
- 691
- Forks
- 202
- Avg merge
- 4h 38m
- Merged PRs (30d)
- 1
Description
**What happened**:
Getting plenty of errors like:
```
2022/01/21 00:39:53 http: TLS handshake error from 10.97.36.76:42820: remote error: tls: bad certificate
```
In my cluster, this IP relates to Prometheus server. Looking at the service configuration I found that the Prometheus scraper annotations were pointing at the webhook TLS port instead of the metrics port.
**What you expected to happen**:
No errors in the logs
**How to reproduce it (as minimally and precisely as possible)**:
Install Prometheus in the cluster where the webhook is running with scrape enabled for service endpoints.
**Anything else we need to know?**:
**Environment**:
- AWS Region: us-east-1
- EKS Platform version (if using EKS, run `aws eks describe-cluster --name --query cluster.platformVersion`): eks4
- Kubernetes version (if using EKS, run `aws eks describe-cluster --name --query cluster.version`): 1.21
- Webhook Version: master branch at `a65cc3d9c61cf6fc43f0f985818c474e0867d786`
Contributor guide
Research direction
Inspect the Kubernetes service configuration and its Prometheus scraper annotations, focusing on the webhook TLS port versus the metrics port. Reproduce with Prometheus scraping enabled for service endpoints and verify that scraping uses the metrics port and the TLS handshake errors no longer appear.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes, prometheus
- Domain
- cloud, devops, observability-sre
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100