storage: a node with a faulty (but not failed) disk causes a cluster availability outage

**Describe the problem**

A node with a faulty disk may not be detected by the disk stall detector. If some IO requests stall or fail, but others succeed, the node can remain in a degraded state indefinitely. This can result in a cluster level availability outage. We don't have any testing simulating this situation or an approach to address it.

Either KV or storage layer should detect and handle this.

**To Reproduce**

This happened on a serverless cluster and caused transient unavailability for ~20 hour until the disk problem was finally detected by the disk stall detector. This impacted all the customers on the cluster during that window.

**Environment:**
- CockroachDB version v24.1.2
- Serverless cluster

Jira issue: CRDB-41602