pingcap / pingcap/tidb

GRPC idle connection recycling and request concurrency, causing tikvRPC to fail

Open
#48,092 0 comments 0 reactions 0 assignees View on GitHub
type/question
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report
Since tikv-client-go is not very active, it is issued here and can be discussed here.

Phenomenon:
When the cluster load is very low, p99 is very high. When the cluster load is high, p99 returns to normal.

Cause Analysis:
When the cluster load is low and a request comes, if atomic.CompareAndSwapUint32(&c.idleNotify, 1, 0) is satisfied, it will run in the background
go c.recycleIdleConnArray(), and this request needs to get the grpc connection to send the request.
That is to say, when a TCP connection is being closed and a request is sent on this connection, in a concurrent situation, this request is likely to fail and trigger backoff, resulting in higher latency.
https://github.com/tikv/client-go/blob/v2.0.7/internal/client/client.go#L592

How to reproduce:
On a cluster, access once at an interval of idleTimeout = 3 * time.Minute, and it must appear.
If accessed every few seconds, it will appear occasionally.

Troubleshooting process:
1) It is found that tikv-client API p99 is relatively high, about 100ms.
![image](https://github.com/pingcap/tidb/assets/4653075/f7bcacef-a308-4a03-ba1f-8675ffecd04f)

2) Find out through opentracing that time is spent on backoff;
![image](https://github.com/pingcap/tidb/assets/4653075/769847a6-2c7b-4423-94ba-db482e743651)

3) Statistics on the metric on tikv-client found that the metric of backoff is the same as the counter of batch_client_reset metric, so the method of recycleIdleConnArray was located.
4) Modify recycleIdleConnArray to run synchronously, and the problem disappears.

Question 1: Solution for recycling idle connection ??
a) Modify recycleIdleConnArray to run synchronously, and you will not encounter connection reset. However,I'm not sure whether it will cause other problems when close a dead connection.
b) Or, change idle connection to daemon task recycling?

Question 2: Is the initial backoff period 100ms appropriate, and can it be confiurable ?
Found out this this configuration affects the latency of long0-tail requests, like p99.999 .
![image](https://github.com/pingcap/tidb/assets/4653075/9694d3b9-6d0b-41a5-9288-2f670144d196)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.