cockroachdb / cockroachdb/cockroach

kvserver: Replica.Send write path ctx cancellation doesn't work when disk stalled

Open
#98,957 2 comments 0 reactions 0 assignees View on GitHub
C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

When the leaseholder's disk is stalled and a client that is waiting for a write has its ctx canceled, we don't return from `Replica.Send`. This is because the call to `abandon` here:

https://github.com/cockroachdb/cockroach/blob/b4216ac71b92c5014d10f84db89f1542033c86f6/pkg/kv/kvserver/replica_raft.go#L307-L310

tries to acquire `raftMu`, but `raftMu` is with near certainty held by the raft processing goroutine for the range which will be stuck on the disk.

**To Reproduce**

https://github.com/cockroachdb/cockroach/pull/98852 TestWriteWithStuckLeaseholderDisk

**Expected behavior**

Ctx cancellation is obeyed.

Jira issue: CRDB-39894

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.