roachtest: decommission/drains/dead failed

roachtest.decommission/drains/dead [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestWeeklyBazel/21429484?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestWeeklyBazel/21429484?buildTab=artifacts#/decommission/drains/dead) on master @ [033407065d004b785922f3199c8fb43f5e225980](https://github.com/cockroachdb/cockroach/commits/033407065d004b785922f3199c8fb43f5e225980):

Failed with:

```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/decommission.go:1115
pkg/util/retry/retry.go:344
pkg/util/retry/retry.go:298
pkg/util/retry/retry.go:343
pkg/cmd/roachtest/tests/decommission.go:1113
pkg/cmd/roachtest/tests/decommission.go:102
pkg/cmd/roachtest/test_runner.go:1853
src/runtime/asm_amd64.s:1771
Error: Received unexpected error:
decommission failed: COMMAND_PROBLEM: exit status 1
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runDecommissionDrains.func2
| pkg/cmd/roachtest/tests/decommission.go:1115
| github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1
| pkg/util/retry/retry.go:344
| github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
| pkg/util/retry/retry.go:298
| github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts
| pkg/util/retry/retry.go:343
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runDecommissionDrains
| pkg/cmd/roachtest/tests/decommission.go:1113
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerDecommission.func3
| pkg/cmd/roachtest/tests/decommission.go:102
| main.(*testRunner).runTest.func2
| pkg/cmd/roachtest/test_runner.go:1853
| runtime.goexit
| src/runtime/asm_amd64.s:1771
Wraps: (2) decommission failed
Wraps: (3) Node 1. Command with error:
| ``````
| ./cockroach node decommission --wait=none --format=csv 4 --port={pgport:1} --certs-dir=certs
| ``````
| stdout:id,is_live,replicas,is_decommissioning,membership,is_draining,readiness,blocking_ranges
| 4,false,7,true,decommissioning,false,allocation errors,2
|
| stderr:ranges blocking decommission detected
| n4 has 2 replicas blocked with error: "1 matching stores are currently throttled: [s3: suspect]"
|
| ERROR: Cannot decommission nodes.
| Failed running "node decommission"
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *hintdetail.withDetail (4) errors.Cmd (5) *exec.ExitError
Test: decommission/drains/dead
(require.go:1401).NoError: FailNow called
test artifacts and logs in: /artifacts/decommission/drains/dead/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21429484-1784358158-37-n4cpu4-0001 | 35.237.103.235 | 10.142.3.8 |
| teamcity-21429484-1784358158-37-n4cpu4-0002 | 104.196.111.152 | 10.142.2.230 |
| teamcity-21429484-1784358158-37-n4cpu4-0003 | 35.229.88.46 | 10.142.2.252 |
| teamcity-21429484-1784358158-37-n4cpu4-0004 | 35.237.87.148 | 10.142.2.123 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- metamorphicLeases=epoch
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21429484/decommission-drains-dead/1784370700870/1784371008111)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21429484-1784358158-37-n4cpu4&from_ts=1784370700870&to_ts=1784371008111&live=false&storage=flex_tier)

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*decommission/drains/dead.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65911