basalt-26.3: roachtest: multitenant/distsql/instances=20/bundle=on/timeout=0 failed

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=0 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21499601?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21499601?buildTab=artifacts#/multitenant/distsql/instances=20/bundle=on/timeout=0) on basalt-26.3 @ [7b7d64a360fe4249bf6409fb266081ab370be355](https://github.com/cockroachdb/cockroach/commits/7b7d64a360fe4249bf6409fb266081ab370be355):

Failed with:

```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/multitenant_distsql.go:159
pkg/cmd/roachtest/tests/multitenant_distsql.go:43
pkg/cmd/roachtest/test_runner.go:1618
src/runtime/asm_amd64.s:1771
Error: "0" is not greater than "0"
Test: multitenant/distsql/instances=20/bundle=on/timeout=0
Messages: All nodes didn't show up in time. EXPLAIN (VEC):
│
├ Node 1
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 2
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 3
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 4
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 5
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ └ *colrpc.Inbox
├ Node 6
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 7
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 9
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 12
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 13
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 14
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 15
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 16
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
└ Node 20
└ *colrpc.Outbox
└ *colexec.UnorderedDistinct
└ *colexec.ParallelUnorderedSynchronizer
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
└ *colflow.routerOutputOp
└ *colflow.HashRouter
└ *colexec.UnorderedDistinct
└ *colfetcher.ColBatchScan
(require.go:620).Greater: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=0/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21499601-1788543659-47-n4cpu4-0001 | | 10.91.98.184 |
| teamcity-21499601-1788543659-47-n4cpu4-0002 | | 10.91.99.127 |
| teamcity-21499601-1788543659-47-n4cpu4-0003 | | 10.91.99.126 |
| teamcity-21499601-1788543659-47-n4cpu4-0004 | | 10.91.99.129 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=expiration
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21499601/multitenant-distsql-instances-20-bundle-on-timeout-0/1788546940344/1788547878440)

_Logs were not uploaded to Datadog_

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant/distsql/instances=20/bundle=on/timeout=0.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)