cockroachdb / cockroachdb/cockroach
roachtest: restore/nodeShutdown/coordinator failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.restore/nodeShutdown/coordinator [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21476924?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21476924?buildTab=artifacts#/restore/nodeShutdown/coordinator) on release-25.4.15-rc @ [4913a926920d3003e1223156615a6a6a7d524b59](https://github.com/cockroachdb/cockroach/commits/4913a926920d3003e1223156615a6a6a7d524b59):
```
(test_runner.go:1622).func1: failed during post test assertions (see test-post-assertions.log): pq: replica unavailable: (n2,s2):12 unable to serve request to r7:/Table/{3-4} [(n1,s1):1, (n2,s2):12, (n4,s4):3, next=13, gen=24]: closed timestamp: 1787313857.511351048,0 (2026-08-21 12:04:17); raft status: {"id":"c","term":7,"vote":"0","commit":447,"lead":"0","leadEpoch":"0","raftState":"StatePreCandidate","applied":447,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
test artifacts and logs in: /artifacts/restore/nodeShutdown/coordinator/cpu_arch=fips/run_1
```
Parameters:
- arch=fips
- cloud=gce
- coverageBuild=false
- cpu=8
- encrypted=false
- fs=ext4
- localSSD=false
- metamorphicLeases=expiration
- runtimeAssertionsBuild=false
- ssd=0
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21476924/restore-nodeshutdown-coordinator/1787312639181/1787315612418)
Same failure on other branches
- #152388 roachtest: restore/nodeShutdown/coordinator failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-3 T-db-server branch-release-25.3.0-rc]
/cc @cockroachlabs/disaster-recovery
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*restore/nodeShutdown/coordinator.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Contributor guide
Research direction
Read pkg/cmd/roachtest/README.md and inspect test_runner.go around line 1622. Start with test-post-assertions.log and the artifacts under /artifacts/restore/nodeShutdown/coordinator/cpu_arch=fips/run_1, then use the linked TeamCity logs and Grafana view to investigate the leaderless replica failure. Done means restore/nodeShutdown/coordinator completes without the replica-unavailable post-test assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100