aws / aws/amazon-eks-pod-identity-webhook

Prometheus scraper misconfigured in the k8s service

Open
#136 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
691
Forks
202
Avg merge
4h 38m
Merged PRs (30d)
1

Description

**What happened**:
Getting plenty of errors like:
```
2022/01/21 00:39:53 http: TLS handshake error from 10.97.36.76:42820: remote error: tls: bad certificate
```
In my cluster, this IP relates to Prometheus server. Looking at the service configuration I found that the Prometheus scraper annotations were pointing at the webhook TLS port instead of the metrics port.

**What you expected to happen**:
No errors in the logs

**How to reproduce it (as minimally and precisely as possible)**:
Install Prometheus in the cluster where the webhook is running with scrape enabled for service endpoints.

**Anything else we need to know?**:

**Environment**:
- AWS Region: us-east-1
- EKS Platform version (if using EKS, run `aws eks describe-cluster --name --query cluster.platformVersion`): eks4
- Kubernetes version (if using EKS, run `aws eks describe-cluster --name --query cluster.version`): 1.21
- Webhook Version: master branch at `a65cc3d9c61cf6fc43f0f985818c474e0867d786`

Contributor guide

Open the contributing guide

Research direction

Inspect the Kubernetes service configuration and its Prometheus scraper annotations, focusing on the webhook TLS port versus the metrics port. Reproduce with Prometheus scraping enabled for service endpoints and verify that scraping uses the metrics port and the TLS handshake errors no longer appear.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes, prometheus
Domain
cloud, devops, observability-sre
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.