Enhance TSO Request Processing to Address Blocking Issues and Improve Reliability
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
### Problem Statement:
In our deployment scenario, TiKV serves as the metadata storage for a large-scale distributed file system with thousands of clients concurrently requesting TSO timestamps through PD followers.
The current implementation in `PDProtoRequest.postProcess()` sequentially sent responses directly to the gRPC stream of thousands of clients using `r.stream.Send(response)`.
Under this high-concurrency workload, when some clients were slow to process responses or experienced network latency, this direct sending mechanism would block the TSO dispatcher goroutine.
This blocking behavior had severe consequences:
1. It caused cascading delays throughout the TSO processing pipeline
2. Led to increased latency for all TSO proxy requests
### Solution:
We introduced a buffered channel for each TSO stream to resolve these issues:
1. Added a buffered channel `tsoRespCh` with capacity 2 in the gRPC service for each TSO stream
2. Modified `PDProtoRequest.postProcess()` to send responses to the channel instead of directly to the stream
3. Implemented a select statement with a default case that drops responses when the channel is full, preventing blocking
These enhancements maintain the non-blocking benefits while improving reliability and observability, especially for our high-concurrency deployment with thousands of concurrent clients.
Contributor guide
Assessment
This issue has not been assessed yet.