kvserver: detect deadlocks, then dump goroutines & fatal

In https://github.com/cockroachdb/cockroach/issues/159139 we observed a store whose raft scheduler had stopped processing events. We weren't able to RCA this, but suspect a deadlock somewhere in KV-land. To improve obs into issues like this,

- [ ] add a mechanism to detect stalls (these [notes](https://cockroachlabs.atlassian.net/wiki/spaces/CORE/pages/edit-v2/5063409728#Summary.1) have some ideas what kind of stalls to detect) including replica and store mutexes, as well as raft scheduler progress
- [ ] land something like https://github.com/cockroachdb/cockroach/pull/159352 that allows triggering goroutine dumps when a stall is detected
- [ ] on detected stall, trigger goroutine dump
- [ ] on prolonged stall, trigger fatal.

We have something similar at the storage level already, so in this issue we would be lifting similar ideas up the stack.

https://github.com/cockroachdb/cockroach/blob/24c746434d12f97b646f1e763bfb4018af17377b/pkg/storage/fs/fs.go#L42-L62

Jira issue: CRDB-57904