GRPC idle connection recycling and request concurrency, causing tikvRPC to fail
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Since tikv-client-go is not very active, it is issued here and can be discussed here.
Phenomenon:
When the cluster load is very low, p99 is very high. When the cluster load is high, p99 returns to normal.
Cause Analysis:
When the cluster load is low and a request comes, if atomic.CompareAndSwapUint32(&c.idleNotify, 1, 0) is satisfied, it will run in the background
go c.recycleIdleConnArray(), and this request needs to get the grpc connection to send the request.
That is to say, when a TCP connection is being closed and a request is sent on this connection, in a concurrent situation, this request is likely to fail and trigger backoff, resulting in higher latency.
https://github.com/tikv/client-go/blob/v2.0.7/internal/client/client.go#L592
How to reproduce:
On a cluster, access once at an interval of idleTimeout = 3 * time.Minute, and it must appear.
If accessed every few seconds, it will appear occasionally.
Troubleshooting process:
1) It is found that tikv-client API p99 is relatively high, about 100ms.

2) Find out through opentracing that time is spent on backoff;

3) Statistics on the metric on tikv-client found that the metric of backoff is the same as the counter of batch_client_reset metric, so the method of recycleIdleConnArray was located.
4) Modify recycleIdleConnArray to run synchronously, and the problem disappears.
Question 1: Solution for recycling idle connection ??
a) Modify recycleIdleConnArray to run synchronously, and you will not encounter connection reset. However,I'm not sure whether it will cause other problems when close a dead connection.
b) Or, change idle connection to daemon task recycling?
Question 2: Is the initial backoff period 100ms appropriate, and can it be confiurable ?
Found out this this configuration affects the latency of long0-tail requests, like p99.999 .

Contributor guide
Assessment
This issue has not been assessed yet.