cockroachdb / cockroachdb/cockroach
cli: TestDebugCheckStore failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
cli.TestDebugCheckStore [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsIbmCloudLinuxS390x_UnitTestsS390x/20654395?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsIbmCloudLinuxS390x_UnitTestsS390x/20654395?buildTab=artifacts#/) on master @ [35ec42c6905a9f6b85c59dcd3a2f9041df2c7858](https://github.com/cockroachdb/cockroach/commits/35ec42c6905a9f6b85c59dcd3a2f9041df2c7858):
```
=== RUN TestDebugCheckStore
test_log_scope.go:171: test logs captured to: /artifacts/tmp/_tmp/0eff02e3d605640c0624f6bcc67cf8d4/logTestDebugCheckStore164741670
test_log_scope.go:82: use -show-logs to present logs inline
testcluster.go:485: deadline below read timestamp is nonsensical; txn has would have no chance to commit. Deadline: 1761595318.322656069,0. Read timestamp: 1761595322.560744779,2 Previous Deadline: 0,0.
panic.go:635: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0eff02e3d605640c0624f6bcc67cf8d4/logTestDebugCheckStore164741670
I251027 20:02:05.693319 1800 2@kv/kvserver/replica_proposal.go:627 [T1,Vsystem,n1,s1,r33/1:/NamespaceTable/{30-Max},raft] 1 new range lease repl=(n1,s1):1 seq=1 start=0,0 type=LeaseLeader term=6 min-exp=1761595331.669083982,0 pro=1761595325.687915288,0 acq=Request promoted from repl=(n1,s1):1 seq=1 start=0,0 type=LeaseExpiration exp=1761595331.669083982,0 pro=1761595325.669083982,0 acq=Request
I251027 20:02:05.707726 1888 2@kv/kvserver/replica_proposal.go:627 [T1,Vsystem,n1,s1,r15/1:/Table/1{2-3},raft] 2 new range lease repl=(n1,s1):1 seq=1 start=0,0 type=LeaseLeader term=6 min-exp=1761595331.679328344,0 pro=1761595325.699896473,0 acq=Request promoted from repl=(n1,s1):1 seq=1 start=0,0 type=LeaseExpiration exp=1761595331.679328344,0 pro=1761595325.679328344,0 acq=Request
I251027 20:02:05.707871 424749 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r15/1:/Table/1{2-3}] 3 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.707949 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r1/1:/{Min-System/NodeL…}] 4 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708020 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r4/1:/System{/tsd-tse}] 5 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708078 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r26/1:/Table/2{3-4}] 6 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708134 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r5/1:/{Systemtse-Table/0}] 7 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708184 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] 8 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708237 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r19/1:/Table/1{6-7}] 9 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708289 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r15/1:/Table/1{2-3}] 10 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708348 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r29/1:/Table/2{6-7}] 11 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708401 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r11/1:/Table/{7-8}] 12 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708451 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r27/1:/Table/2{4-5}] 13 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708501 73760 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] 14 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
I251027 20:02:05.708743 424915 13@kv/kvserver/replicate_queue.go:859 [T1,Vsystem,n1,replicate,s1,r33/1:/NamespaceTable/{30-Max}] 15 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster
--- FAIL: TestDebugCheckStore (63.80s)
```
Help
See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)
/cc @cockroachdb/kv-triage @cockroachdb/server
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestDebugCheckStore.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-55941
Contributor guide
Research direction
Start with TestDebugCheckStore and the failure output from the linked TeamCity build, then inspect testcluster.go:485 where the deadline error is reported. Use the saved test logs and artifacts to identify why the test deadline falls below the read timestamp; done means the test passes reliably on the affected s390x configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100