Lower log level for failed RPC attempt with a successful retry
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Feature Description
Failed RPC calls are currently logged at log level `ERROR`.
https://github.com/hashicorp/consul/blob/51549bd23283f2fdd0d2ad5fa63fa569bc5da33f/agent/consul/client.go#L317
The error log is written even though the RPC call might succeed via a retry. The result is that the log file is filled with errors that looks a bit scary even though the RPC has probably been retried and completed successfully. It would be nice to clarify if it is the whole RPC call that failed or if it is just the current rpc call attempt that failed and that there is x number of retries going to happen before giving up completely.
#### Use Case(s)
We have been tuning our performance parameters according to the documentation in order to reduce the number of 500 errors seen during cluster leader re-election. We have not been able to get rid of the errors completely which caused us to look into the source code which seems to log the error for each failed rpc attempt. A more detailed log message would improve monitoring during cluster upgrades and other types of troubleshooting.
Contributor guide
Assessment
This issue has not been assessed yet.