Azure / Azure/azure-workload-identity

workload identity webhook controller pod show `http: TLS handshake error from`

Open
#781 8 comments 3 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
337
Forks
122
Avg merge
1h 7m
Merged PRs (30d)
3

Description

**Describe the bug**
When there are heavy load to webhook, the webhook would report `http: TLS handshake error from <>: EOF` message .

**Steps To Reproduce**
Create a deployment with 2000 pods that enables workload identity.

```
...
2023/03/06 08:24:03 http: TLS handshake error from 10.224.54.106:54844: EOF
2023/03/06 08:24:03 http: TLS handshake error from 10.224.54.106:54846: EOF
2023/03/06 08:24:03 http: TLS handshake error from 10.224.54.106:54862: EOF
2023/03/06 08:24:03 http: TLS handshake error from 10.224.54.6:60372: EOF
2023/03/06 08:24:03 http: TLS handshake error from 10.224.54.6:60398: EOF
2023/03/06 08:24:04 http: TLS handshake error from 10.224.54.6:60396: EOF
2023/03/06 08:24:04 http: TLS handshake error from 10.224.54.106:54884: EOF
2023/03/06 08:24:04 http: TLS handshake error from 10.224.54.6:60446: EOF
2023/03/06 08:24:04 http: TLS handshake error from 10.224.54.106:54876: EOF
2023/03/06 08:24:04 http: TLS handshake error from 10.224.54.6:60392: EOF
2023/03/06 08:24:04 http: TLS handshake error from 10.224.54.6:60412: EOF
2023/03/06 08:24:04 http: TLS handshake error from 10.224.54.6:60388: EOF
2023/03/06 08:24:05 http: TLS handshake error from 10.224.54.106:54898: EOF
2023/03/06 08:24:05 http: TLS handshake error from 10.224.54.6:60422: EOF
...
```

**Expected behavior**
Do not show such TLS error

**Logs**

**Environment**

- Kubernetes version (use `kubectl version`):
- Cloud provider or hardware configuration:
- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Install tools:
- Network plugin and version (if this is a network-related bug):
- Others:

**Additional context**

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with a deployment of 2,000 pods that enables workload identity, then inspect the workload identity webhook controller logs for repeated TLS handshake errors. Done means the webhook no longer reports these TLS errors under heavy load.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
authentication, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.