cockroachdb / cockroachdb/cockroach
kvserver: requests stuck for 24 hours
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
I started a 12 node cluster on master to import tpcc 500k to produce a fixture. After a couple days the cluster still claimed to be working on the last import job but appeared idle. On closer inspection, some `AddSSTable` had been pending for over 24 hours.
Separately, on this cluster it was observed that `SHOW RANGES` consistently returns an error:
```
root@localhost:26257/defaultdb> select count(*) from crdb_internal.ranges_no_leases;
ERROR: replica unavailable: (n4,s4):4VOTER_DEMOTING_LEARNER unable to serve request to r79947:/Table/114/1/1{18467/5/-3001/1-25000/1/-3001/1} [(n9,s9):1, (n4,s4):4VOTER_DEMOTING_LEARNER, (n5,s5):3, (n2,s2):5VOTER_INCOMING, next=6, gen=11520, sticky=1711830222.241516817,0]: closed timestamp: 1711829727.822144851,0 (2024-03-30 20:15:27); raft status: {"id":"4","term":7,"vote":"4","commit":29,"lead":"4","raftState":"StateLeader","applied":29,"progress":{"5":{"match":0,"next":26,"state":"StateSnapshot"},"1":{"match":0,"next":26,"state":"StateSnapshot"},"3":{"match":63899,"next":63900,"state":"StateReplicate"},"4":{"match":63899,"next":63900,"state":"StateReplicate"}},"leadtransferee":"0"}: encountered poisoned latch /Local/RangeID/79947/r/AbortSpan/"4f56911b-8cb1-471e-ab86-7c2823016e4d"@0,0
```
[1.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816438/1.stacks.txt)
[2.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816439/2.stacks.txt)
[6.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816443/6.stacks.txt)
[7.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816444/7.stacks.txt)
[8.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816445/8.stacks.txt)
[9.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816447/9.stacks.txt)
[10.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816448/10.stacks.txt)
[11.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816449/11.stacks.txt)
[12.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816450/12.stacks.txt)
[3.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816440/3.stacks.txt)
[4.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816441/4.stacks.txt)
[5.stacks.txt](https://github.com/cockroachdb/cockroach/files/14816442/5.stacks.txt)
[1.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816473/1.problem-ranges.json)
[2.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816474/2.problem-ranges.json)
[3.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816475/3.problem-ranges.json)
[4.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816476/4.problem-ranges.json)
[5.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816477/5.problem-ranges.json)
[6.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816478/6.problem-ranges.json)
[7.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816479/7.problem-ranges.json)
[8.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816480/8.problem-ranges.json)
[9.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816481/9.problem-ranges.json)
[10.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816482/10.problem-ranges.json)
[11.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816483/11.problem-ranges.json)
[12.problem-ranges.json](https://github.com/cockroachdb/cockroach/files/14816484/12.problem-ranges.json)
Jira issue: CRDB-37254
Contributor guide
Assessment
This issue has not been assessed yet.