cockroachdb / cockroachdb/cockroach

basalt-26.3: roachtest: multitenant/distsql/instances=20/bundle=off/timeout=1 failed

Open
#174,437 7 comments 0 reactions 0 assignees View on GitHub
B-runtime-assertions-enabled basalt-green branch-basalt-26.3 C-test-failure O-roachtest O-robot release-blocker T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.multitenant/distsql/instances=20/bundle=off/timeout=1 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21493566?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21493566?buildTab=artifacts#/multitenant/distsql/instances=20/bundle=off/timeout=1) on basalt-26.3 @ [1326f194d41c7a5187da0509cee33d331d2229de](https://github.com/cockroachdb/cockroach/commits/1326f194d41c7a5187da0509cee33d331d2229de):

Failed with:

```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/multitenant_distsql.go:159
pkg/cmd/roachtest/tests/multitenant_distsql.go:43
pkg/cmd/roachtest/test_runner.go:1558
src/runtime/asm_amd64.s:1771
Error: "0" is not greater than "0"
Test: multitenant/distsql/instances=20/bundle=off/timeout=1
Messages: All nodes didn't show up in time. EXPLAIN (VEC):
│
├ Node 1
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colexec.UnorderedDistinct
│ │ └ *colexec.ParallelUnorderedSynchronizer
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colflow.routerOutputOp
│ │ │ └ *colflow.HashRouter
│ │ │ └ *colexec.UnorderedDistinct
│ │ │ └ *colfetcher.ColBatchScan
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ ├ *colrpc.Inbox
│ │ └ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 2
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 3
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 4
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 7
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 9
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ └ *colrpc.Inbox
├ Node 10
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 13
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 15
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 16
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 17
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 18
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 19
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colflow.routerOutputOp
│ └ *colflow.HashRouter
│ └ *colexec.UnorderedDistinct
│ └ *colfetcher.ColBatchScan
└ Node 20
└ *colrpc.Outbox
└ *colexec.UnorderedDistinct
└ *colexec.ParallelUnorderedSynchronizer
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colflow.routerOutputOp
│ └ *colflow.HashRouter
│ └ *colexec.UnorderedDistinct
│ └ *colfetcher.ColBatchScan
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
└ *colrpc.Inbox
(require.go:620).Greater: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=off/timeout=1/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21493566-1788241958-46-n4cpu4-0001 | | 10.91.98.142 |
| teamcity-21493566-1788241958-46-n4cpu4-0002 | | 10.91.98.139 |
| teamcity-21493566-1788241958-46-n4cpu4-0003 | | 10.91.98.133 |
| teamcity-21493566-1788241958-46-n4cpu4-0004 | | 10.91.98.125 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicDRPC=true
- metamorphicLeases=default
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21493566/multitenant-distsql-instances-20-bundle-off-timeout-1/1788245197507/1788246078827)

_Logs were not uploaded to Datadog_

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant/distsql/instances=20/bundle=off/timeout=1.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/tests/multitenant_distsql.go at lines 43 and 159, then inspect the linked TeamCity log and artifacts for the assertion-enabled run. Determine why the 20-instance multitenant DistSQL test reports that all nodes did not appear before the timeout; done requires reproducing or explaining the failure and restoring a passing test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.