The configuration/implementation of async gRPC may be not optimal

Enhancement

Configuration Issues

https://github.com/pingcap/tiflash/blob/d7e4e2995cebf0cf25e13a2a91adebada9390b66/dbms/src/Interpreters/Settings.h#L363-L364
The async gRPC server uses these configurations for EstablishMPPConnection.
async_pollers_per_cq is the number of threads per completion queue. (default 200)
async_cqs is the number of completion queues. (default 1)
The number of gRPC threads is async_pollers_per_cq * async_cqs = 200.

In my opinion, these default configurations have two issues.

1. The default thread number is too large. In view of the fact that async gRPC uses a non-blocking socket, in theory, there is no blocking point when doing RPC work. Therefore, the thread number should not exceed the number of CPU cores.
2. The default completion queue number is too small, which may introduce some unnecessary synchronization overheads.
   For example, CqEventQueue uses an MPSC queue and it also uses a spinlock to support multiple consumers. In addition,
   pollset_work is called when calling CompletionQueue::Next. A mutex is acquired during this call. Although sometimes this mutex is released, this overhead can not be ignored especially when the thread number is 200.

Actually, the official guide of gRPC performance says

If having to use the async completion-queue API, the best scalability trade-off is having numcpu’s threads. The ideal number of completion queues in relation to the number of threads can change over time (as gRPC C++ evolves), but as of gRPC 1.41 (Sept 2021), using 2 threads per completion queue seems to give the best performance.

At present, TiFlash uses v1.26 gRPC. The perf_notes from v.126 gRPC says

Right now, the best performance trade-off is having numcpu's threads and one completion queue per thread.

I guess using multiple threads per completion queue is good for load balance but the number should not be too large. We can carefully test it to gain the best performance.

Implementation Issues

The first issue is about notify_cq.
https://github.com/pingcap/tiflash/blob/d7e4e2995cebf0cf25e13a2a91adebada9390b66/dbms/src/Server/FlashGrpcServerHolder.cpp#L141-L157
From the code above, we can see there is a another gRPC thread pool for notify_cq. In fact, the default number gRPC thread for EstablishMPPConnection is 400(200 for cq, 200 for notify_cq), which is a scary number.

What is the difference between call_cq and notification_cq in gRPC?

Notification_cq gets the tag back indicating a call has started. All subsequent operations (reads, writes, etc) on that call report back to call_cq. For most async servers my recommendation is to use the same cq.
This allows fine-grained control over which threads handle which kinds of events (based on which queues they are polling). Like you may have a master thread polling the notification_cq and worker threads all polling their own call_cqs, or something like that.

This code tests when the notify_cq and call_cq is called.

I think we do not need to control which threads handle notification events so the notify_cq should be the same as call_cq then the default 200 threads can be removed totally.

By the way, grpc-rs also uses one completion queue both for call_cq and notify_cq. code here

The second issue is about combining different gRPC thread pools. Async gRPC client in TiFlash also has a gRPC thread pool.
https://github.com/pingcap/tiflash/blob/d7e4e2995cebf0cf25e13a2a91adebada9390b66/dbms/src/Server/Server.cpp#L1242-L1248 (good to see that its pool size is std::thread::hardware_concurrency).
Combining different gRPC thread pools can reduce the thread number and context switch. This also makes it easier to add new async RPC in the future.
It can be done with some class abstraction and refactor.

先从 dbms/src/Interpreters/Settings.h 中的异步 gRPC 设置开始，然后阅读 dbms/src/Server/FlashGrpcServerHolder.cpp 和 dbms/src/Server/Server.cpp 中的服务器设置。查看链接的 gRPC v1.26 性能说明，并测量候选的 completion-queue 和 thread-pool 配置。配置和池设计得到性能结果的支持，并且提议的重构经过验证，即表示完成。