cockroachdb / cockroachdb/cockroach

cli: TestDebugCheckStore failed

Open
#156,328 11 comments 0 reactions 0 assignees View on GitHub
branch-master C-test-failure O-robot P-3 s390x-test-failure T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

cli.TestDebugCheckStore [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsIbmCloudLinuxS390x_UnitTestsS390x/20654395?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsIbmCloudLinuxS390x_UnitTestsS390x/20654395?buildTab=artifacts#/) on master @ [35ec42c6905a9f6b85c59dcd3a2f9041df2c7858](https://github.com/cockroachdb/cockroach/commits/35ec42c6905a9f6b85c59dcd3a2f9041df2c7858):

```
=== RUN TestDebugCheckStore
test_log_scope.go:171: test logs captured to: /artifacts/tmp/_tmp/0eff02e3d605640c0624f6bcc67cf8d4/logTestDebugCheckStore164741670
test_log_scope.go:82: use -show-logs to present logs inline
testcluster.go:485: deadline below read timestamp is nonsensical; txn has would have no chance to commit. Deadline: 1761595318.322656069,0. Read timestamp: 1761595322.560744779,2 Previous Deadline: 0,0.
panic.go:635: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0eff02e3d605640c0624f6bcc67cf8d4/logTestDebugCheckStore164741670
I251027 20:02:05.693319 1800 2@kv/kvserver/replica_proposal.go:627 [T1,Vsystem,n1,s1,r33/1:/NamespaceTable/{30-Max},raft] 1 new range lease repl=(n1,s1):1 seq=1 start=0,0 type=LeaseLeader term=6 min-exp=1761595331.669083982,0 pro=1761595325.687915288,0 acq=Request promoted from repl=(n1,s1):1 seq=1 start=0,0 type=LeaseExpiration exp=1761595331.669083982,0 pro=1761595325.669083982,0 acq=Request
I251027 20:02:05.707726 1888 2@kv/kvserver/replica_proposal.go:627 [T1,Vsystem,n1,s1,r15/1:/Table/1{2-3},raft] 2 new range lease repl=(n1,s1):1 seq=1 start=0,0 type=LeaseLeader term=6 min-exp=1761595331.679328344,0 pro=1761595325.699896473,0 acq=Request promoted from repl=(n1,s1):1 seq=1 start=0,0 type=LeaseExpiration exp=1761595331.679328344,0 pro=1761595325.679328344,0 acq=Request
I251027 20:02:05.707871 424749 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r15/1:/Table/1{2-3}] 3 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.707949 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r1/1:/{Min-System/NodeL…}] 4 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708020 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r4/1:/System{/tsd-tse}] 5 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708078 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r26/1:/Table/2{3-4}] 6 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708134 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r5/1:/{Systemtse-Table/0}] 7 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708184 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] 8 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708237 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r19/1:/Table/1{6-7}] 9 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708289 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r15/1:/Table/1{2-3}] 10 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708348 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r29/1:/Table/2{6-7}] 11 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708401 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r11/1:/Table/{7-8}] 12 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708451 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r27/1:/Table/2{4-5}] 13 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708501 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] 14 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708743 424915 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r33/1:/NamespaceTable/{30-Max}] 15 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
--- FAIL: TestDebugCheckStore (63.80s)
```
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachdb/kv-triage @cockroachdb/server

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestDebugCheckStore.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-55941

Contributor guide

Open the contributing guide

Research direction

Start with TestDebugCheckStore and the failure output from the linked TeamCity build, then inspect testcluster.go:485 where the deadline error is reported. Use the saved test logs and artifacts to identify why the test deadline falls below the read timestamp; done means the test passes reliably on the affected s390x configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.