hashicorp / hashicorp/consul

Consul HTTPS API - OpenSSL SSL_write: Broken pipe, errno 32 (AKA Your IP is issuing too many concurrent connections, please rate limit your calls)

Open
#19,622 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

Due to Consul HTTPS API service failing -> `(curl: (16) OpenSSL SSL_write: Broken pipe, errno 32)`
but HTTP API gives proper error `Your IP is issuing too many concurrent connections, please rate limit your calls`

can HTTPS API also provide this error instead of breaking the pipe?

initial issue: tasks failing to start due to envoy proxy failing

example logs:
```
cat ./55141183-81f8-4536-850f-e1b033fe76ec/alloc/logs/envoy_bootstrap.stderr.0

failed fetch proxy config from local agent: Get "https://127.0.0.1:8501/v1/agent/service/_nomad-task-55141183-81f8-4536-850f-e1b033fe76ec-emo-sidecar-proxy": write tcp 127.0.0.1:37612->127.0.0.1:8501: write: broken pipe
```

troubleshooting on CLI
```
curl -k https://127.0.0.1:8501/v1/agent/service/_nomad-task-55141183-81f8-4536-850f-e1b033fe76ec-emo-sidecar-proxy
curl: (16) OpenSSL SSL_write: Broken pipe, errno 32
```

however, these are all looking fine
- `cli` `consul members`
- `cli` `consul catalog datacenters`
- `cli` `consul catalog nodes`
- `cli` `consul catalog services`
- `UI` `node health`

So there should be something odd with 8501 on consul causing it to malfunction.

consul-logs:
- nothing to be seen ... not even on debug.

and then for some reason not clear to me was trying http, which gave:
```
curl -k http://127.0.0.1:8500/v1/agent/service/_nomad-task-55141183-81f8-4536-850f-e1b033fe76ec-emo-sidecar-proxy
Your IP is issuing too many concurrent connections, please rate limit your calls
```
~~so ... solution then would be adding the limits.http_max_conns_per_client (but that's not my bug/request)~~

my bug/request would be: please inform more clearly that this is the issue
and not just fail the https service

or if really needed to fail the https service,
log that somewhere, perhaps every x seconds aggregated, that you dropped x connections from y
but preferable not silent drops, as this seems legitimate traffic.

---

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 45
services = 51
build:
prerelease =
revision = 68f81912
version = 1.16.2
version_metadata =
consul:
acl = enabled
known_servers = 5
server = false
runtime:
arch = amd64
cpu_count = 16
goroutines = 1120
max_procs = 16
os = linux
version = go1.20.8
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 55
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 13669
members = 15
query_queue = 0
query_time = 1
```

### Operating system and Environment details
- Ubuntu 22.04.3 LTS

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.