Metastore raft cluster does not self-heal after single-node disk write stall — requires manual pod restart

#### Describe the bug

In a 3-node metastore raft cluster, a transient write-latency spike on a single node's PVC
triggered the `timeoutLogStore` write timeout (added in #4892/#4935), causing that node to
step down as leader. Instead of the cluster electing a new leader and recovering within
seconds, the affected node got stuck reporting `node is not the leader` / failing readiness
probes for several minutes, and other components (ingesters, compaction workers) kept
round-robining requests into the stuck node via `metastore-client`. The cluster only
recovered once the affected pod was manually restarted — raft did not self-heal on its own
despite 2 of 3 nodes remaining healthy throughout.

This looks like a real-world hit of the mechanism described in #5307
("fix(metastore): fix follower livelock from abandoned log-store writes", currently
draft/unmerged): the timeout wrapper aborts the *wait* on a slow `StoreLogs` call but can't
cancel the underlying write, which keeps running in the background and can still land later,
violating raft-wal's single-writer/monotonicity invariants and leaving the node in a
half-broken "zombie" state.

#### To Reproduce

1. Start Pyroscope metastore (version/SHA:
grafana/pyroscope:2.2.0), 3-node raft cluster, each backed by its own PVC.
2. Induce a write-latency spike on one node's PVC sufficient to exceed the metastore's
`LogStoreTimeout` (default 10s) on a `StoreLogs` call — in our case this occurred
organically from underlying storage contention, not a fault injection.
3. Observe the affected node log `log store write timed out after 10s` and step down as
leader (`raft: failed to commit logs`).
4. Observe kubelet readiness probes fail on that pod, and other components see
`rpc error: code = Unavailable desc = node is not the leader` while round-robining across
all 3 metastore servers.

#### Expected behavior

The two unaffected nodes should elect a new leader and the cluster should return to a healthy
state within a normal raft election window (single-digit seconds), even if the node that
experienced the disk stall takes longer to fully recover. Downstream clients should converge
on the new leader quickly rather than repeatedly retrying the stuck node.

Instead, in our incident:
- Metastore-0 and metastore-2 recovered readiness within seconds of the leadership change.
- Metastore-1 (the node that experienced the stall) continued failing readiness probes
roughly every 10s for **~3.5 more minutes**, and continued being selected by
`metastore-client`'s random leader-selection logic on other components' retries.
- The cluster only fully recovered after metastore-1's pod was manually restarted.

#### Environment

- Infrastructure: Kubernetes (RKE2, Hetzner-backed bare-metal workers)
- Deployment tool: Helm
- Metastore: 3 replicas, each with a dedicated PVC (StatefulSet `pyroscope-metastore`)

#### Additional Context

Timeline of the incident (all times UTC, 2026-07-28):

| Time | Event |
|---|---|
| 06:09:00–06:11:00 | Average write latency on the affected node's PVC spikes from a ~2.3ms baseline to 183–225ms (2-min average) — consistent with an underlying multi-second write stall |
| 06:09:59–06:10:02 | Kubelet readiness probes fail (`context deadline exceeded`) on all three metastore pods |
| 06:10:04 | `msg="log store write timed out after 10s"` → `raft: failed to commit logs` → affected node steps down as leader |
| 06:10:04–~06:13:28 | The two unaffected nodes recover readiness within seconds; the affected node alone continues failing readiness probes every ~10s |
| ~06:13:00 | Affected pod restarted manually; cluster recovers immediately |

Node-level health (`DiskPressure`, `MemoryPressure`, `Ready` conditions, node-exporter scrape
continuity) stayed clean throughout — this was an isolated storage-layer event on one node's
volume, not a node eviction/pressure event.

Representative log lines:
level=error caller=raftnode msg="failed to commit logs" error="log store write timed out after 10s"
level=warn caller=grpc_logging.go method=/raft_node.RaftNodeService/ReadIndex duration=10.000998033s err="rpc error: code = Unavailable desc = node is not the leader"
level=error caller=methods.go component=metastore-client msg="metastore client error" err="rpc error: code = Unavailable desc = node is not the leader" server_id=pyroscope-metastore-1...
level=info caller=methods.go component=metastore-client msg="changing metastore client leader" current=pyroscope-metastore-0... new=pyroscope-metastore-1...

Related:
- #5307 — draft fix for the abandoned-write livelock mechanism; this issue may be a good
real-world trace to validate it against.
- `pkg/metastore/client/methods.go`'s `selectInstance()` has no way to temporarily deprioritize
a server that just errored — it always re-selects uniformly at random among all servers,
which likely extended the practical outage window for downstream callers beyond what the
raft layer alone caused.
- #5257 — same error strings but a different root cause (single-replica v2.0.3 regression),
not a duplicate.

Start by reading the timeoutLogStore behavior described here and the draft fix in #5307, then inspect pkg/metastore/client/methods.go, especially selectInstance(). Reproduce the single-node PVC write stall in a 3-node Kubernetes metastore and compare raft readiness and client selection with the expected election-window recovery. Done means the healthy nodes recover while the stalled node is isolated, and downstream retries stop repeatedly selecting the failing server.

Written by the indexing model from the issue text.