cockroachdb / cockroachdb/cockroach
cli: mark rpc context as poisoned during disk stall handling
Open
A-server-start-drain
A-storage
C-bug
T-storage
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
In #96145, we adapted our disk stall handling to shutdown all open sockets so the process appears dead to the rest of the cluster, even if the process is unable to terminate because some threads performing disk I/O are stuck in uninterruptible sleep within the kernel. It's possible for retry mechanisms to open new outgoing connections after all sockets are shutdown. To prevent this, we should consider marking the rpc context as poisoned before closing all outgoing connections. Connection opening code would check the poison marker before opening new connections.
Jira issue: CRDB-24077
Contributor guide
Assessment
This issue has not been assessed yet.