tikv / tikv/pd

Enhance TSO Request Processing to Address Blocking Issues and Improve Reliability

Open
#9,662 0 comments 0 reactions 0 assignees View on GitHub
contribution type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

### Problem Statement:

In our deployment scenario, TiKV serves as the metadata storage for a large-scale distributed file system with thousands of clients concurrently requesting TSO timestamps through PD followers.

The current implementation in `PDProtoRequest.postProcess()` sequentially sent responses directly to the gRPC stream of thousands of clients using `r.stream.Send(response)`.
Under this high-concurrency workload, when some clients were slow to process responses or experienced network latency, this direct sending mechanism would block the TSO dispatcher goroutine.

This blocking behavior had severe consequences:
1. It caused cascading delays throughout the TSO processing pipeline
2. Led to increased latency for all TSO proxy requests

### Solution:

We introduced a buffered channel for each TSO stream to resolve these issues:

1. Added a buffered channel `tsoRespCh` with capacity 2 in the gRPC service for each TSO stream
2. Modified `PDProtoRequest.postProcess()` to send responses to the channel instead of directly to the stream
3. Implemented a select statement with a default case that drops responses when the channel is full, preventing blocking

These enhancements maintain the non-blocking benefits while improving reliability and observability, especially for our high-concurrency deployment with thousands of concurrent clients.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.