kvserver: store decommissioning

**Describe the problem**

This issue tracks issues identified during multi-store testing https://docs.google.com/document/d/1WTQx-JgD0myf9oAAUKq44e-nTXMD_DqUsKdZwX4hvYU/edit?tab=t.0 (thanks to Tobi and Arul for running the experiments).

- [x] Panic occurred when stopping a node and removing
- `roachprod ssh $c:5 -- mv /mnt/data3/cockroach{,.old}`
```
E250328 01:32:38.248327 97 1@util/log/logcrash/crash_reporting.go:192 ⋮ [T1,Vsystem,n?] 13 +‹directory "/mnt/data3/cockroach/auxiliary/wals-among-stores" may contain relevant WALs›
```
- work around: removing the wal_recovery section in OPTIONS files on other stores on n1
- [ ] No underreplicated ranges because leaseholder reports this based on unavailable follower nodes not unavailable stores.
- [ ] Log spam on nodes with the following and some other StoreNotFound error handling. Error handling has been fixed by https://github.com/cockroachdb/cockroach/pull/143673. But log spam lines still seem to be there - need more confirmation.
```
W250328 01:38:18.507624 932917 kv/kvserver/store_raft.go:663 ⋮ [T1,Vsystem,n2,s4] 6595 raft error: node 1 claims to not contain store 3 for replica (n1,s3):?: store 3 was not found

W250328 01:38:18.507689 932902 kv/kvserver/raft_transport.go:1067 ⋮ [T1,Vsystem,n2] 6596 while processing outgoing Raft queue to node 1: store 3 was not found:
W250328 01:38:19.507293 932967 kv/kvserver/store_raft.go:663 ⋮ [T1,Vsystem,n2,s4] 6597 raft error: node 1 claims to not contain store 3 for replica (n1,s3):?: store 3 was not found
```
- [ ] Some code mixes up until store dead with time until node dead. Example: https://github.com/cockroachdb/cockroach/blob/13afd78424202cf5bf6e634d061a7f82540d602b/pkg/kv/kvserver/liveness/liveness.go#L40-L58.
- [x] When starting again with fresh `/mnt/data3`,

```
E250328 02:12:00.929664 95 1@util/log/logcrash/crash_reporting.go:192 ⋮ [T1,Vsystem,n?] 13 +‹directory "/mnt/data1/cockroach/auxiliary/wals-among-stores" may contain relevant WALs›
```
happens again. Removing WAL failover sections from /mnt/data[12]/cockroach/OPTIONS* fixed it. But `/mnt/data3` ends up being store 16.
- [ ] We need to be able to operate on node store pairs when enqueuing range in a replica queue. There can be replicas with same range id on the same node but different stores now. Also check other places where we take range id as an input.
- [ ] A unit test with multi store where we manually rebalance from two stores of same node would be a good idea.
- [ ] When moving replicas off a store via zone configs `constraints = '[-store6]'`,
- Footgun: The attrs roachprod assigns are based on the “store index plus 1” (basically the N in /mnt/dataN) so there is no s6.
- After fixing above, replica count doesn’t drop proactively, but enqueueing in the replicate queue does work. Over ten minutes, it dropped a bit. Random errors like below:
```
delegated_sender: term:10 first_index:121 sender_queue_name:REPLICATE_QUEUE descriptor_generation:322 queue_on_delegate_len:-1 snap_id:4c43e4dc-
4796-4033-bab0-6458639e8e47 ›: grpc: ‹giving up during snapshot reservation due to cluster setting "kv.snapshot_receiver.reservation_queue_timeout_fraction": context deadline exceeded› [code 4/DeadlineExceeded]
```
- Another unrelated store n5 is withdrawing support from itself.
```
kv/kvserver/storeliveness/supporter_state.go:321 ⋮ [T1,Vsystem,n5,s14] 11284 withdrew support for store {NodeID:5 StoreID:15}; old = {Target:{NodeID:5 StoreID:15} Epoch:13 Expiration:1743131589.206
424449,0}, new = {Target:{NodeID:5 StoreID:15} Epoch:14 Expiration:0,0}
```
- Turns out that this was just because they were saturating the bandwidth with `select count(*) from bank.bank;`. But this seems to have worked fairly well when the store was just dead.

Jira issue: CRDB-50684

Epic CRDB-48087