hashicorp / hashicorp/consul

Lower log level for failed RPC attempt with a successful retry

Open
#8,000 0 comments 1 reaction 0 assignees View on GitHub
theme/operator-usability type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Feature Description

Failed RPC calls are currently logged at log level `ERROR`.

https://github.com/hashicorp/consul/blob/51549bd23283f2fdd0d2ad5fa63fa569bc5da33f/agent/consul/client.go#L317

The error log is written even though the RPC call might succeed via a retry. The result is that the log file is filled with errors that looks a bit scary even though the RPC has probably been retried and completed successfully. It would be nice to clarify if it is the whole RPC call that failed or if it is just the current rpc call attempt that failed and that there is x number of retries going to happen before giving up completely.

#### Use Case(s)

We have been tuning our performance parameters according to the documentation in order to reduce the number of 500 errors seen during cluster leader re-election. We have not been able to get rid of the errors completely which caused us to look into the source code which seems to log the error for each failed rpc attempt. A more detailed log message would improve monitoring during cluster upgrades and other types of troubleshooting.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.