cockroachdb / cockroachdb/cockroach

roachtest: restore/nodeShutdown/coordinator failed

Open
#173,691 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery branch-release-25.4.15-rc C-test-failure O-roachtest O-robot release-blocker T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.restore/nodeShutdown/coordinator [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21476924?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21476924?buildTab=artifacts#/restore/nodeShutdown/coordinator) on release-25.4.15-rc @ [4913a926920d3003e1223156615a6a6a7d524b59](https://github.com/cockroachdb/cockroach/commits/4913a926920d3003e1223156615a6a6a7d524b59):

```
(test_runner.go:1622).func1: failed during post test assertions (see test-post-assertions.log): pq: replica unavailable: (n2,s2):12 unable to serve request to r7:/Table/{3-4} [(n1,s1):1, (n2,s2):12, (n4,s4):3, next=13, gen=24]: closed timestamp: 1787313857.511351048,0 (2026-08-21 12:04:17); raft status: {"id":"c","term":7,"vote":"0","commit":447,"lead":"0","leadEpoch":"0","raftState":"StatePreCandidate","applied":447,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
test artifacts and logs in: /artifacts/restore/nodeShutdown/coordinator/cpu_arch=fips/run_1
```

Parameters:
- arch=fips
- cloud=gce
- coverageBuild=false
- cpu=8
- encrypted=false
- fs=ext4
- localSSD=false
- metamorphicLeases=expiration
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21476924/restore-nodeshutdown-coordinator/1787312639181/1787315612418)

Same failure on other branches

- #152388 roachtest: restore/nodeShutdown/coordinator failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot P-3 T-db-server branch-release-25.3.0-rc]

/cc @cockroachlabs/disaster-recovery

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*restore/nodeShutdown/coordinator.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Read pkg/cmd/roachtest/README.md and inspect test_runner.go around line 1622. Start with test-post-assertions.log and the artifacts under /artifacts/restore/nodeShutdown/coordinator/cpu_arch=fips/run_1, then use the linked TeamCity logs and Grafana view to investigate the leaderless replica failure. Done means restore/nodeShutdown/coordinator completes without the replica-unavailable post-test assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.