receive 429 even if the outstanding connections are below threshold

#### Overview of the Issue

We have default settings for the `http_max_conns_per_client`, which is 200. However, after a network outage in some of our clusters, we've start to see 429, `Your IP is issuing too many concurrent connections` even tho the outstanding connections are below the threashold.

List of connections

```
root [ /home/photonconfig ]# curl localhost:8500
Your IP is issuing too many concurrent connections, please rate limit your calls
root [ /home/photonconfig ]# netstat -anp | grep 8500
tcp 0 0 127.0.0.1:43156 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43118 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43180 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:37342 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43164 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43182 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43178 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43184 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43134 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:37334 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43158 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43116 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:60640 127.0.0.1:8500 ESTABLISHED 63986/telegraf
tcp 0 0 127.0.0.1:43132 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43198 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43126 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43140 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:36252 127.0.0.1:8500 ESTABLISHED 62241/traefik
tcp 0 0 127.0.0.1:43148 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43142 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43146 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43172 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43160 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43124 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:37286 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43196 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43186 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:60880 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43138 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:37320 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43168 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43200 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43194 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:60952 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43130 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43162 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43176 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43110 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43166 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43122 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43152 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43170 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43114 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43150 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43136 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43128 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43144 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43108 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:37340 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43120 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:43174 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp 0 0 127.0.0.1:37346 127.0.0.1:8500 ESTABLISHED 98384/nomad
tcp6 0 0 :::8500 :::* LISTEN 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45148 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43138 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45100 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43186 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44854 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43130 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43196 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45158 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44882 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45078 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43166 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43162 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43118 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45270 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43136 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43116 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43180 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:36252 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43200 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45048 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43144 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43198 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43108 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43176 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:37320 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:60952 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44800 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43194 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45242 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45296 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45084 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43182 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45000 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43152 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43160 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43172 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44874 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45166 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44894 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43184 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44944 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45316 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43124 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43110 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43122 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:37334 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45302 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43126 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43134 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43140 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:37286 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45066 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:60640 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43150 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44974 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43148 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44908 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43168 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45020 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43128 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44788 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:60880 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45172 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:37340 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43114 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43146 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45238 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45130 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44918 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43142 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43156 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43174 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43158 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45250 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:37346 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:37342 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43120 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43164 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44958 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43178 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45342 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45346 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43170 ESTABLISHED 98192/consul
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44900 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44796 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:45180 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44952 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44842 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:44890 TIME_WAIT -
tcp6 0 0 127.0.0.1:8500 127.0.0.1:43132 ESTABLISHED 98192/consul
root [ /home/photonconfig ]# netstat -anp | grep 8500 | grep ESTABLISHED | wc -l
104
```

#### Reproduction Steps

Not exactly sure how to reproduce the issue, but what we know:

1. there were network issues on the cluster, causes a lot rpc failures: `agent.client: RPC failed to server: method=ConnectCA.Roots server=x.x.x.x:8300 error="rpc error making call: rpc error making call: EOF"`
1. we have consul mesh setup with nomad jobs (not sure if this matters)

### Consul info for both Client and Server

Client info

```
Error querying agent: Unexpected response code: 429 (Your IP is issuing too many concurrent connections, please rate limit your calls
)
```

Server info

```
agent:
check_monitors = 1
check_ttls = 0
checks = 3
services = 5
build:
prerelease =
revision = 6530cf37
version = 1.9.2
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.102.7.52:8300
server = true
raft:
applied_index = 8715830
commit_index = 8715830
fsm_pending = 0
last_contact = 16.203371ms
last_log_index = 8715830
last_log_term = 7
last_snapshot_index = 8710095
last_snapshot_term = 7
latest_configuration = [{Suffrage:Voter ID:742178de-e910-4d33-1332-944bb032efa7 Address:10.102.7.53:8300} {Suffrage:Voter ID:d2d81602-9d59-12d5-0c97-ebc756e166e9 Address:10.102.7.51:8300} {Suffrage:Voter ID:ded403f2-c036-6a0a-9551-cbdfd66fe939 Address:10.102.7.52:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 7
runtime:
arch = amd64
cpu_count = 2
goroutines = 709
max_procs = 2
os = linux
version = go1.15.6
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 7
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 19165
members = 30
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 19
members = 3
query_queue = 0
query_time = 1
```

### Operating system and Environment details

Photon OS, `Linux us02pL-csl1 4.19.182-1.ph3 #1-photon SMP Thu Mar 25 20:47:18 UTC 2021 x86_64 GNU/Linux`