tikv / tikv/pd

TestMicroserviceTSOClientSuite/TestGetMinTS is flaky

Open
#11,113 2 comments 0 reactions 0 assignees View on GitHub
needs-type
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Flaky Test

`TestMicroserviceTSOClientSuite/TestGetMinTS`

### Which jobs are failing

`pull-unit-test-next-gen-3`

### CI link

https://prow.tidb.net/view/gs/prow-tidb-logs/pr-logs/pull/tikv_pd/11100/pull-unit-test-next-gen-3/2084945640642580480

### Reason for failure (if possible)

The failed subtest reported `none of tso server/pod responded` and `can't query all the tso keyspace groups. Asked 0/1, expected 3` at `tests/integrations/tso/client_test.go:315`. During the same window, TSO allocator logs reported about 1.4 seconds of clock offset, longer than the one-second timeout used when PD queries each TSO server.

The suite was executed by `pd-ut`, which randomly shuffles all top-level test tasks with the process-random `math/rand/v2` source before feeding them to 24 parallel workers. This changes the concurrent test mix on every CI run. `TestGetMinTS` generates 7,500 `GetMinTS` calls and 37,500 `GetTS` calls, so an unlucky concurrent task mix can delay TSO responses beyond the fixed deadline.

The suite-order hypothesis was checked separately: `TestDiscoverTSOServiceWithLegacyPath`, the only suite method preceding `TestGetMinTS`, followed by `TestGetMinTS` passed five consecutive race-enabled runs. `TestGetMinTS` also passed under `GOMAXPROCS=1` and in 12 concurrent local processes. These results show that local resource pressure is a trigger but not a sufficient standalone cause; the confirmed run-to-run nondeterminism is the randomized `pd-ut` task queue.

The proposed fix is to sort tasks by package and test name before enqueueing them, preserving parallelism while making each shard's task order reproducible.

### Anything else

The failure was observed on PR #11100, but the failing integration test and randomized runner are unrelated to that PR's client GC changes.

Contributor guide

Open the contributing guide

Research direction

Start with tests/integrations/tso/client_test.go around line 315 and trace how pd-ut shuffles and enqueues top-level test tasks. Reproduce the flaky TestMicroserviceTSOClientSuite/TestGetMinTS behavior under the randomized runner, then verify that task ordering is reproducible by package and test name while retaining parallel workers.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.