basalt-26.3: roachtest: cluster_creation failed

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21508877?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21508877?buildTab=artifacts#/sqlsmith/setup=empty/setting=default) on basalt-26.3 @ [edc226e9d007d4a208c831af595863f67e923ebe](https://github.com/cockroachdb/cockroach/commits/edc226e9d007d4a208c831af595863f67e923ebe):

```
time=2026-09-10T12:31:29.998Z level=DEBUG msg="starting SSH command" cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 command="mkdir -p -m 700 /tmp/roachmgr-install && tar -xzf - -C /tmp/roachmgr-install && sudo bash /tmp/roachmgr-install/install.sh && rm -rf /tmp/roachmgr-install" host=10.91.96.181:22
time=2026-09-10T12:31:30.053Z level=DEBUG msg="Creating system users..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
time=2026-09-10T12:31:30.062Z level=DEBUG msg="Creating system users..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:30.068Z level=DEBUG msg="Creating system users..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
time=2026-09-10T12:31:30.080Z level=DEBUG msg="Creating directories..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
time=2026-09-10T12:31:30.086Z level=DEBUG msg="Installing certificates..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
time=2026-09-10T12:31:30.093Z level=DEBUG msg="Creating directories..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:30.098Z level=DEBUG msg="Creating directories..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
time=2026-09-10T12:31:30.100Z level=DEBUG msg="Installing certificates..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:30.103Z level=DEBUG msg="Installing binaries..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
time=2026-09-10T12:31:30.105Z level=DEBUG msg="Installing certificates..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
time=2026-09-10T12:31:30.108Z level=DEBUG msg="Fetching roachmgr binary..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
time=2026-09-10T12:31:30.116Z level=DEBUG msg="Installing binaries..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:30.121Z level=DEBUG msg="Fetching roachmgr binary..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:30.122Z level=DEBUG msg="Installing binaries..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
time=2026-09-10T12:31:30.127Z level=DEBUG msg="Fetching roachmgr binary..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
time=2026-09-10T12:31:30.503Z level=DEBUG msg="2026-09-10 12:31:30 URL:https://storage.googleapis.com/basalt-public/roachmgr/v0.1.14/roachmgr-v0.1.14.linux-amd64.tgz [63680796/63680796] -> \"/tmp/artifact-3460.tgz\" [1]" cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
time=2026-09-10T12:31:30.514Z level=DEBUG msg="2026-09-10 12:31:30 URL:https://storage.googleapis.com/basalt-public/roachmgr/v0.1.14/roachmgr-v0.1.14.linux-amd64.tgz [63680796/63680796] -> \"/tmp/artifact-3518.tgz\" [1]" cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
time=2026-09-10T12:31:30.566Z level=DEBUG msg="2026-09-10 12:31:30 URL:https://storage.googleapis.com/basalt-public/roachmgr/v0.1.14/roachmgr-v0.1.14.linux-amd64.tgz [63680796/63680796] -> \"/tmp/artifact-3516.tgz\" [1]" cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:32.142Z level=DEBUG msg="Installing config and service files..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
time=2026-09-10T12:31:32.167Z level=DEBUG msg="Installing config and service files..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
time=2026-09-10T12:31:32.170Z level=DEBUG msg="Capping journald disk usage..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
time=2026-09-10T12:31:32.192Z level=DEBUG msg="Installing config and service files..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:32.195Z level=DEBUG msg="Capping journald disk usage..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
time=2026-09-10T12:31:32.219Z level=DEBUG msg="Capping journald disk usage..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:32.333Z level=DEBUG msg="Starting Roachmgr agent..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
time=2026-09-10T12:31:32.359Z level=DEBUG msg="Starting Roachmgr agent..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
time=2026-09-10T12:31:32.367Z level=DEBUG msg="Starting Roachmgr agent..." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:33.043Z level=DEBUG msg="Created symlink /etc/systemd/system/multi-user.target.wants/roachmgr-root.service → /etc/systemd/system/roachmgr-root.service." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
time=2026-09-10T12:31:33.066Z level=DEBUG msg="Created symlink /etc/systemd/system/multi-user.target.wants/roachmgr-root.service → /etc/systemd/system/roachmgr-root.service." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
time=2026-09-10T12:31:33.077Z level=DEBUG msg="Created symlink /etc/systemd/system/multi-user.target.wants/roachmgr-root.service → /etc/systemd/system/roachmgr-root.service." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:33.768Z level=DEBUG msg="Created symlink /etc/systemd/system/multi-user.target.wants/roachmgr.service → /etc/systemd/system/roachmgr.service." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
time=2026-09-10T12:31:33.797Z level=DEBUG msg="Created symlink /etc/systemd/system/multi-user.target.wants/roachmgr.service → /etc/systemd/system/roachmgr.service." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:33.799Z level=DEBUG msg="Created symlink /etc/systemd/system/multi-user.target.wants/roachmgr.service → /etc/systemd/system/roachmgr.service." cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
time=2026-09-10T12:31:34.532Z level=DEBUG msg="Installation complete" cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001 host=10.91.96.198:22
time=2026-09-10T12:31:34.532Z level=DEBUG msg="Installation complete" cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002 host=10.91.96.223:22
10.91.96.198:9090: installed
time=2026-09-10T12:31:34.536Z level=INFO msg="10.91.96.198:9090: installed"
10.91.96.223:9090: installed
time=2026-09-10T12:31:34.537Z level=INFO msg="10.91.96.223:9090: installed"
time=2026-09-10T12:31:34.560Z level=DEBUG msg="Installation complete" cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt machine_count=3 machine=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003 host=10.91.96.181:22
10.91.96.181:9090: installed
time=2026-09-10T12:31:34.564Z level=DEBUG msg="connecting via SSH" cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt addr=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001.roachprod.crdb.dev:22 user=ubuntu
time=2026-09-10T12:31:34.564Z level=INFO msg="10.91.96.181:9090: installed"
time=2026-09-10T12:31:34.564Z level=DEBUG msg="SSH client pool event" cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt event=ssh_client_pool_hit addr=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001.roachprod.crdb.dev:22 user=ubuntu
time=2026-09-10T12:31:34.564Z level=DEBUG msg="creating file archive via SSH" cluster=teamcity-21508877-1789020668-255-n4cpu4-basalt operation=cluster_apply operation_id=6032ad8d-24ec-409c-bc27-90e4f347c18b infra=teamcity-21508877-1789020668-255-n4cpu4-basalt path=/var/lib/roachmgr/certs host=10.91.96.198:22
time=2026-09-10T12:31:34.632Z level=DEBUG msg="creating gRPC client connection" addr=10.91.96.198:9090
time=2026-09-10T12:33:34.538Z level=DEBUG msg="SSH client pool event" event=ssh_client_pool_ttl_expiration addr=teamcity-21508877-1789020668-255-n4cpu4-basalt-0002.roachprod.crdb.dev:22 user=ubuntu
time=2026-09-10T12:33:34.564Z level=DEBUG msg="SSH client pool event" event=ssh_client_pool_ttl_expiration addr=teamcity-21508877-1789020668-255-n4cpu4-basalt-0003.roachprod.crdb.dev:22 user=ubuntu
time=2026-09-10T12:33:34.632Z level=DEBUG msg="SSH client pool event" event=ssh_client_pool_ttl_expiration addr=teamcity-21508877-1789020668-255-n4cpu4-basalt-0001.roachprod.crdb.dev:22 user=ubuntu: context deadline exceeded [owner=test-eng]
```

Parameters:
- MetamorphicBasaltMultiAZ=false
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- runtimeAssertionsBuild=true
- storageNodesVersion=v0.1.19-nightly.20260909
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*cluster_creation.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Start with the TeamCity log and artifacts for roachtest.cluster_creation at commit edc226e9d007d4a208c831af595863f67e923ebe. Trace the cluster_apply failure and compare it with a run without assertions enabled. Done means the cluster_creation roachtest completes successfully and the failure is covered by an appropriate regression test.

Written by the indexing model from the issue text.