cockroachdb / cockroachdb/cockroach

basalt-26.3: roachtest: multitenant/distsql/instances=20/bundle=off/timeout=1 failed

Open
#174,179 2 comments 0 reactions 0 assignees View on GitHub
branch-basalt-26.3 C-test-failure O-roachtest O-robot T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

roachtest.multitenant/distsql/instances=20/bundle=off/timeout=1 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RedRoachtestNightlyGcePrivateBasaltBazel/21489828?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RedRoachtestNightlyGcePrivateBasaltBazel/21489828?buildTab=artifacts#/multitenant/distsql/instances=20/bundle=off/timeout=1) on basalt-26.3 @ [bdb2b728a10ba1855108bff17d3c777e0f0625a1](https://github.com/cockroachdb/cockroach/commits/bdb2b728a10ba1855108bff17d3c777e0f0625a1):

Failed with:

```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/multitenant_distsql.go:159
pkg/cmd/roachtest/tests/multitenant_distsql.go:43
pkg/cmd/roachtest/test_runner.go:1558
src/runtime/asm_amd64.s:1771
Error: "0" is not greater than "0"
Test: multitenant/distsql/instances=20/bundle=off/timeout=1
Messages: All nodes didn't show up in time. EXPLAIN (VEC):
│
├ Node 1
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colexec.UnorderedDistinct
│ │ └ *colexec.ParallelUnorderedSynchronizer
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colflow.routerOutputOp
│ │ │ └ *colflow.HashRouter
│ │ │ └ *colexec.UnorderedDistinct
│ │ │ └ *colfetcher.ColBatchScan
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ └ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 2
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 3
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 4
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 7
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 8
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 9
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 10
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 11
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 12
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colflow.routerOutputOp
│ └ *colflow.HashRouter
│ └ *colexec.UnorderedDistinct
│ └ *colfetcher.ColBatchScan
├ Node 13
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 14
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 16
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 17
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 18
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ └ *colrpc.Inbox
└ Node 19
└ *colrpc.Outbox
└ *colexec.UnorderedDistinct
└ *colexec.ParallelUnorderedSynchronizer
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colflow.routerOutputOp
│ └ *colflow.HashRouter
│ └ *colexec.UnorderedDistinct
│ └ *colfetcher.ColBatchScan
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
└ *colrpc.Inbox
(require.go:620).Greater: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=off/timeout=1/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21489828-1787896812-34-n4cpu4-0001 | | 10.91.98.129 |
| teamcity-21489828-1787896812-34-n4cpu4-0002 | | 10.91.99.63 |
| teamcity-21489828-1787896812-34-n4cpu4-0003 | | 10.91.111.192 |
| teamcity-21489828-1787896812-34-n4cpu4-0004 | | 10.91.98.128 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=default
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21489828/multitenant-distsql-instances-20-bundle-off-timeout-1/1787898239338/1787899143194)

_Logs were not uploaded to Datadog_

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant/distsql/instances=20/bundle=off/timeout=1.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67480

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/tests/multitenant_distsql.go, especially lines 43 and 159, then inspect the linked TeamCity log and artifacts for the basalt-26.3 run. Reproduce multitenant/distsql/instances=20/bundle=off/timeout=1 and determine why all nodes do not appear within the timeout; done means the roachtest completes without this assertion failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.