cockroachdb / cockroachdb/cockroach

cli: mark rpc context as poisoned during disk stall handling

Open
#96,342 2 comments 0 reactions 0 assignees View on GitHub
A-server-start-drain A-storage C-bug T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

In #96145, we adapted our disk stall handling to shutdown all open sockets so the process appears dead to the rest of the cluster, even if the process is unable to terminate because some threads performing disk I/O are stuck in uninterruptible sleep within the kernel. It's possible for retry mechanisms to open new outgoing connections after all sockets are shutdown. To prevent this, we should consider marking the rpc context as poisoned before closing all outgoing connections. Connection opening code would check the poison marker before opening new connections.

Jira issue: CRDB-24077

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.