Consul stop answer on 8500 port when many wait(blocking) api queries are made
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
After upgrade from consul 1.5.3 to 1.6.7 we found that from some time, on agents, where we making many waiting(blocking) api queries, consul agent stop answer on 8500 port. We made about 50-100 blocking queries simultaneously(not too much). Not any interesting in logs just `EOF` error on api calls and RPC communications to servers, consul doesn't crush at this moment, just simple stop answer on 8500 port, and doesn't eat CPU or memory(looks like in dead block)
to demonstrate EOF erros we provide a little bit error log from buggy consul agent, other log messages are the same
```
2020/09/07 20:47:39 [ERR] consul: "Health.ServiceNodes" RPC failed to server 172.29.20.160:8300: rpc error making call: EOF
2020/09/07 20:47:39 [ERR] http: Request GET /v1/health/service/self-cert?index=181938153&token=, error: rpc error making call: EOF from=127.0.0.1:47592
2020/09/07 20:47:39 [ERR] consul: "KVS.Get" RPC failed to server 172.29.20.160:8300: rpc error making call: EOF
2020/09/07 20:47:39 [ERR] http: Request GET /v1/kv/app/appsexternalproxy/it-absence-calendar-frontend?index=176490858&token=, error: rpc error making call: EOF from
2020/09/07 20:47:39 [ERR] consul: "KVS.Get" RPC failed to server 172.29.20.160:8300: rpc error making call: EOF
2020/09/07 20:47:39 [ERR] http: Request GET /v1/kv/app/appsexternalproxy/it-eis-gateway?index=138275516&token=, error: rpc error making call: EOF from=127.0.0.1:476
2020/09/07 20:47:39 [ERR] consul: "KVS.Get" RPC failed to server 172.29.20.160:8300: rpc error making call: EOF
2020/09/07 20:47:39 [ERR] http: Request GET /v1/kv/app/appsexternalproxy/web.playrix.com?index=174412054&token=, error: rpc error making call: EOF from=127.0.0.1:60
2020/09/07 20:47:39 [ERR] consul: "KVS.Get" RPC failed to server 172.29.20.160:8300: rpc error making call: EOF
2020/09/07 20:47:39 [ERR] http: Request GET /v1/kv/app/appsexternalproxy/it-eis-frontend-reports?index=176491062&token=, error: rpc error making call: EOF from=127.
2020/09/07 20:47:39 [ERR] consul: "Health.ServiceNodes" RPC failed to server 172.29.20.160:8300: rpc error making call: EOF
2020/09/07 20:47:39 [ERR] http: Request GET /v1/health/service/it-eis-voki-gateway?index=181938151&token=, error: rpc error making call: EOF from=127.0.0.1:13858
2020/09/07 20:47:39 [ERR] consul: "KVS.Get" RPC failed to server 172.29.20.160:8300: rpc error making call: EOF
2020/09/07 20:47:39 [ERR] http: Request GET /v1/kv/app/appsexternalproxy/hackathon-2020.playrix.com?index=144594085&token=, error: rpc error making call: EOF from=1
2020/09/07 20:47:39 [ERR] consul: "Health.ServiceNodes" RPC failed to server 172.29.20.160:8300: rpc error making call: EOF
2020/09/07 20:47:39 [ERR] http: Request GET /v1/health/service/hr-http-api?index=182020363&token=, error: rpc error making call: EOF from=127.0.0
```
We need some time to reproduce this on our test stand to provide additional info. For now we just fallback to consul 1.5.3 on client nodes
Contributor guide
Assessment
This issue has not been assessed yet.