hashicorp / hashicorp/consul

Investigation: grpc WARN logs "addrConn.createTransport failed to connect to..."

Open
#17,842 5 comments 3 reactions 0 assignees View on GitHub
theme/internals type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

https://github.com/hashicorp/consul/issues/10603 has been addressed by https://github.com/hashicorp/consul/pull/15701 but users are still reporting seeing WARN logs such as
```
2023-02-06T05:50:55.844+0100 [WARN] agent: [core]grpc: addrConn.createTransport failed to connect to {dc1-10.192.0.5:8300 consul-3 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp ->10.192.0.5:8300: operation was canceled". Reconnecting...
2023-02-06T05:57:43.781+0100 [WARN] agent: [core]grpc: addrConn.createTransport failed to connect to {dc1-10.192.0.3:8300 consul-1.dc1 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp ->10.192.0.3:8300: operation was canceled". Reconnecting...
2023-02-06T05:58:10.466+0100 [WARN] agent: [core]grpc: addrConn.createTransport failed to connect to {dc1-10.192.0.4:8300 consul-2.dc1 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp ->10.192.0.4:8300: operation was canceled". Reconnecting...
2023-02-06T06:50:09.474+0100 [WARN] agent: [core]grpc: addrConn.createTransport failed to connect to {dc1-10.192.0.4:8300 consul-2.dc1 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp ->10.192.0.4:8300: operation was canceled". Reconnecting...
2023-02-06T07:13:16.970+0100 [WARN] agent: [core]grpc: addrConn.createTransport failed to connect to {dc1-10.192.0.4:8300 consul-2.dc1 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp ->10.192.0.4:8300: operation was canceled". Reconnecting...
```

These logs are, as far as the reports show, not as regular as https://github.com/hashicorp/consul/issues/10603 and likely require separate investigation.

It is worth noting that https://github.com/hashicorp/consul/pull/15892 has completely replaced Consul's grpc balancer internals. The WARN logs containing `operation was canceled` are likely (based on previous investigation) caused by updating a subconnection in quick succession, causing a CONNECTING subconnection to close.

These WARN logs should have no impact on Consul; they are a side-effect of periodic server-shuffling and error-handling. It is still inconvenient for operators to deal with WARN-level logs which cannot be easily suppressed.

#### Reproduction Steps

If you see these WARN logs frequently, please provide DEBUG logs for a single server agent (aggregated logs make it difficult to parse).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.