cockroachdb / cockroachdb/cockroach

basalt-26.3: roachtest: multitenant/distsql/instances=20/bundle=on/timeout=1 failed

Open
#174,440 6 comments 0 reactions 0 assignees View on GitHub
B-runtime-assertions-enabled basalt-green branch-basalt-26.3 C-test-failure O-roachtest O-robot release-blocker T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.multitenant/distsql/instances=20/bundle=on/timeout=1 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21493566?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21493566?buildTab=artifacts#/multitenant/distsql/instances=20/bundle=on/timeout=1) on basalt-26.3 @ [1326f194d41c7a5187da0509cee33d331d2229de](https://github.com/cockroachdb/cockroach/commits/1326f194d41c7a5187da0509cee33d331d2229de):

Failed with:

```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/multitenant_distsql.go:159
pkg/cmd/roachtest/tests/multitenant_distsql.go:43
pkg/cmd/roachtest/test_runner.go:1558
src/runtime/asm_amd64.s:1771
Error: "0" is not greater than "0"
Test: multitenant/distsql/instances=20/bundle=on/timeout=1
Messages: All nodes didn't show up in time. EXPLAIN (VEC):
│
├ Node 1
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 2
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 3
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 4
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 5
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 7
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ └ *colrpc.Inbox
├ Node 10
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 11
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 12
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 14
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 15
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 16
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colflow.routerOutputOp
│ └ *colflow.HashRouter
│ └ *colexec.UnorderedDistinct
│ └ *colfetcher.ColBatchScan
├ Node 17
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 18
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
├ Node 19
│ └ *colrpc.Outbox
│ └ *colexec.UnorderedDistinct
│ └ *colexec.ParallelUnorderedSynchronizer
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colflow.routerOutputOp
│ │ └ *colflow.HashRouter
│ │ └ *colexec.UnorderedDistinct
│ │ └ *colfetcher.ColBatchScan
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ ├ *colrpc.Inbox
│ └ *colrpc.Inbox
└ Node 20
└ *colrpc.Outbox
└ *colexec.UnorderedDistinct
└ *colexec.ParallelUnorderedSynchronizer
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colflow.routerOutputOp
│ └ *colflow.HashRouter
│ └ *colexec.UnorderedDistinct
│ └ *colfetcher.ColBatchScan
├ *colrpc.Inbox
├ *colrpc.Inbox
├ *colrpc.Inbox
└ *colrpc.Inbox
(require.go:620).Greater: FailNow called
test artifacts and logs in: /artifacts/multitenant/distsql/instances=20/bundle=on/timeout=1/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21493566-1788241958-48-n4cpu4-0001 | | 10.91.96.85 |
| teamcity-21493566-1788241958-48-n4cpu4-0002 | | 10.91.97.67 |
| teamcity-21493566-1788241958-48-n4cpu4-0003 | | 10.91.97.68 |
| teamcity-21493566-1788241958-48-n4cpu4-0004 | | 10.91.97.64 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=default
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21493566/multitenant-distsql-instances-20-bundle-on-timeout-1/1788245473610/1788246359259)

_Logs were not uploaded to Datadog_

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant/distsql/instances=20/bundle=on/timeout=1.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/tests/multitenant_distsql.go, especially lines 43 and 159, and inspect the linked TeamCity artifacts for the failing run. Reproduce multitenant/distsql/instances=20/bundle=on/timeout=1 and compare assertion-enabled and non-assertion results. Done means the underlying failure is identified and the roachtest passes reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.