tikv / tikv/pd

TSO consistency test does not complete PD leader handoff

Open
#11,219 1 comment 0 reactions 0 assignees View on GitHub
type/ci
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Flaky Test

`TestLegacyTSOConsistencySuite/TestRequestTSOConcurrently` is unstable after
resigning the PD leader.

The test keeps a direct gRPC client connected to the original PD leader. It
then waits only for PD leader discovery before immediately sending concurrent
TSO requests. Depending on election timing, the original endpoint may no
longer be the leader, or the new leader embedded TSO allocator may not be
initialized yet.

Observed failures include:

- `generate tso maximum number of retries exceeded` in #11217:
https://github.com/tikv/pd/actions/runs/34431762649/job/102728565408
- The same subtest also failed during validation of #11147.

Expected behavior: after the leader transition is complete, the consistency
test should connect to the actual leader and start requests only after its
embedded TSO allocator is ready.

Proposed fix: use the normal leader-resign path, wait for the embedded TSO
allocator readiness condition, and reconnect the direct test client to the
elected leader.

Contributor guide

Open the contributing guide

Research direction

Start with TestLegacyTSOConsistencySuite/TestRequestTSOConcurrently and run the failing consistency test to reproduce the handoff race. Trace the existing PD leader-resign path and the embedded TSO allocator readiness condition, then verify that the direct client reconnects to the elected leader and concurrent TSO requests complete reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.