tikv / tikv/pd

TestGetGCStateRejectsOldLeaderAfterTransfer is flaky

Open
#10,956 0 comments 0 reactions 0 assignees View on GitHub
type/ci
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Flaky Test

### Which jobs are failing

```
POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 12956 lock 0xc0012ae5d0
../../cluster.go:175 tests.(*TestServer).Run { s.Lock() } <<<<<
../../cluster.go:174 tests.(*TestServer).Run { func (s *TestServer) Run() error { }
../../cluster.go:682 tests.RunServer.func1 { resC := make(chan error) }

Have been trying to lock it again for more than 30s
goroutine 111 lock 0xc0012ae5d0
../../cluster.go:252 tests.(*TestServer).State { s.RLock() } <<<<<
../../cluster.go:251 tests.(*TestServer).State { func (s *TestServer) State() int32 { }
../../cluster.go:734 tests.(*TestCluster).runInitialServersWithRetry { for _, s := range servers { }
gc_test.go:61 gc.newGCStateLeaderTransitionCluster { re.NoError(err) }
gc_test.go:61 gc.newGCStateLeaderTransitionCluster { re.NoError(err) }
gc_test.go:510 gc.TestGetGCStateRejectsOldLeaderAfterTransfer { re := require.New(t) }

Here is what goroutine 12956 doing now
goroutine 12956 [select, 1 minutes]:
github.com/tikv/pd/server.(*Server).startEtcd(0xc001301808, {0x7821728, 0xc000294eb0})
/home/prow/go/src/github.com/tikv/pd/server/server.go:398 +0x4af
github.com/tikv/pd/server.(*Server).Run(0xc001301808)
/home/prow/go/src/github.com/tikv/pd/server/server.go:678 +0x1af
github.com/tikv/pd/tests.(*TestServer).Run(0xc0012ae5d0)
/home/prow/go/src/github.com/tikv/pd/tests/cluster.go:180 +0x1c7
github.com/tikv/pd/tests.RunServer.func1()
/home/prow/go/src/github.com/tikv/pd/tests/cluster.go:683 +0x45
created by github.com/tikv/pd/tests.RunServer in goroutine 111
/home/prow/go/src/github.com/tikv/pd/tests/cluster.go:683 +0x13a
```

### CI link

https://prow.tidb.net/view/gs/prow-tidb-logs/pr-logs/pull/tikv_pd/10641/pull-unit-test-next-gen-2/2071729564228259840

### Reason for failure (if possible)

### Anything else

Package: github.com/tikv/pd/tests/integrations/gc
Commit: cfef6f4c892e

Contributor guide

Open the contributing guide

Research direction

Start with gc_test.go, especially newGCStateLeaderTransitionCluster and TestGetGCStateRejectsOldLeaderAfterTransfer, then run the named test repeatedly. Trace the lock paths in tests/cluster.go around TestServer.Run, State, and runInitialServersWithRetry, and inspect server/server.go:398 for the blocked startEtcd call. Done means the test no longer intermittently deadlocks in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.