Incubating Project: TiDB over RoCE

Project Incubating Request

TiDB-over-RoCE

Describe the project you want to incubate:

Currently, message transmissions inside TiDB such as raft heartbeat RPCs are based on TCP. Specifically, TiDB server and PD using grpc-go , while TiKV using gRPC Core wrapped by gRPC-rs, both of these gRPC projects using TCP as the underlying communication protocol.

RDMA(Remote Direct Memory Access) allows one machine to read or write to the memory of another machine without requiring any processor cycles on the target. It has been paid more attention since its especial advantages such as Kernel-Bypass and Zero-Copy. As a RDMA protocol based on Ethernet, RoCEv2 (RDMA over Converged Ethernet v2) achieves a very well balance between efficiency of network communication and compatibility in existing data center infrastructure.

This proposal proposes running TiDB over RoCE protocol, which means all communications between tidb server, pd and tikv can benefit from RDMA technology, and higher performance and lower CPU workload are expected.

Estimated Time

10 months

Initial Team Members

张培承(raygift)
李毅伦(typedefstruct32)

Your RFC/Proposal?

The proposal names TiDB server, PD, TiKV, grpc-go, gRPC Core/gRPC-rs, and RoCEv2, but no repository files or tests. Start by reviewing the communication paths described and the RFC/proposal requirements. A completion target is not specified beyond running communications over RoCE with expected performance and CPU benefits.