Consul HTTPS API - OpenSSL SSL_write: Broken pipe, errno 32 (AKA Your IP is issuing too many concurrent connections, please rate limit your calls)
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
Due to Consul HTTPS API service failing -> `(curl: (16) OpenSSL SSL_write: Broken pipe, errno 32)`
but HTTP API gives proper error `Your IP is issuing too many concurrent connections, please rate limit your calls`
can HTTPS API also provide this error instead of breaking the pipe?
initial issue: tasks failing to start due to envoy proxy failing
example logs:
```
cat ./55141183-81f8-4536-850f-e1b033fe76ec/alloc/logs/envoy_bootstrap.stderr.0
failed fetch proxy config from local agent: Get "https://127.0.0.1:8501/v1/agent/service/_nomad-task-55141183-81f8-4536-850f-e1b033fe76ec-emo-sidecar-proxy": write tcp 127.0.0.1:37612->127.0.0.1:8501: write: broken pipe
```
troubleshooting on CLI
```
curl -k https://127.0.0.1:8501/v1/agent/service/_nomad-task-55141183-81f8-4536-850f-e1b033fe76ec-emo-sidecar-proxy
curl: (16) OpenSSL SSL_write: Broken pipe, errno 32
```
however, these are all looking fine
- `cli` `consul members`
- `cli` `consul catalog datacenters`
- `cli` `consul catalog nodes`
- `cli` `consul catalog services`
- `UI` `node health`
So there should be something odd with 8501 on consul causing it to malfunction.
consul-logs:
- nothing to be seen ... not even on debug.
and then for some reason not clear to me was trying http, which gave:
```
curl -k http://127.0.0.1:8500/v1/agent/service/_nomad-task-55141183-81f8-4536-850f-e1b033fe76ec-emo-sidecar-proxy
Your IP is issuing too many concurrent connections, please rate limit your calls
```
~~so ... solution then would be adding the limits.http_max_conns_per_client (but that's not my bug/request)~~
my bug/request would be: please inform more clearly that this is the issue
and not just fail the https service
or if really needed to fail the https service,
log that somewhere, perhaps every x seconds aggregated, that you dropped x connections from y
but preferable not silent drops, as this seems legitimate traffic.
---
### Consul info for both Client and Server
Client info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 45
services = 51
build:
prerelease =
revision = 68f81912
version = 1.16.2
version_metadata =
consul:
acl = enabled
known_servers = 5
server = false
runtime:
arch = amd64
cpu_count = 16
goroutines = 1120
max_procs = 16
os = linux
version = go1.20.8
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 55
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 13669
members = 15
query_queue = 0
query_time = 1
```
### Operating system and Environment details
- Ubuntu 22.04.3 LTS
Contributor guide
Assessment
This issue has not been assessed yet.