TestMicroserviceTSOClientSuite/TestGetMinTS is flaky
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Flaky Test
`TestMicroserviceTSOClientSuite/TestGetMinTS`
### Which jobs are failing
`pull-unit-test-next-gen-3`
### CI link
https://prow.tidb.net/view/gs/prow-tidb-logs/pr-logs/pull/tikv_pd/11100/pull-unit-test-next-gen-3/2084945640642580480
### Reason for failure (if possible)
The failed subtest reported `none of tso server/pod responded` and `can't query all the tso keyspace groups. Asked 0/1, expected 3` at `tests/integrations/tso/client_test.go:315`. During the same window, TSO allocator logs reported about 1.4 seconds of clock offset, longer than the one-second timeout used when PD queries each TSO server.
The suite was executed by `pd-ut`, which randomly shuffles all top-level test tasks with the process-random `math/rand/v2` source before feeding them to 24 parallel workers. This changes the concurrent test mix on every CI run. `TestGetMinTS` generates 7,500 `GetMinTS` calls and 37,500 `GetTS` calls, so an unlucky concurrent task mix can delay TSO responses beyond the fixed deadline.
The suite-order hypothesis was checked separately: `TestDiscoverTSOServiceWithLegacyPath`, the only suite method preceding `TestGetMinTS`, followed by `TestGetMinTS` passed five consecutive race-enabled runs. `TestGetMinTS` also passed under `GOMAXPROCS=1` and in 12 concurrent local processes. These results show that local resource pressure is a trigger but not a sufficient standalone cause; the confirmed run-to-run nondeterminism is the randomized `pd-ut` task queue.
The proposed fix is to sort tasks by package and test name before enqueueing them, preserving parallelism while making each shard's task order reproducible.
### Anything else
The failure was observed on PR #11100, but the failing integration test and randomized runner are unrelated to that PR's client GC changes.
Contributor guide
Research direction
Start with tests/integrations/tso/client_test.go around line 315 and trace how pd-ut shuffles and enqueues top-level test tasks. Reproduce the flaky TestMicroserviceTSOClientSuite/TestGetMinTS behavior under the randomized runner, then verify that task ordering is reproducible by package and test name while retaining parallel workers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100