cockroachdb / cockroachdb/cockroach

roachtest: costfuzz/seed-multi-region failed

Open
#175,575 0 comments 0 reactions 0 assignees View on GitHub
branch-master C-test-failure O-roachtest O-robot O-rsg T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.costfuzz/seed-multi-region [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21520406?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21520406?buildTab=artifacts#/costfuzz/seed-multi-region) on master @ [5d7eb2e1fee29048322d2533eccd0362ba67c0a8](https://github.com/cockroachdb/cockroach/commits/5d7eb2e1fee29048322d2533eccd0362ba67c0a8):

Failed with:

```
(query_comparison_util.go:501).runOneRoundQueryComparison: internal error while running perturbed statement. 447 statements run: pq: internal error: unhandled op distribute
test artifacts and logs in: /artifacts/costfuzz/seed-multi-region/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21520406-1789624655-60-n9cpu4-geo-0001 | | 10.91.97.111 |
| teamcity-21520406-1789624655-60-n9cpu4-geo-0002 | | 10.91.128.62 |
| teamcity-21520406-1789624655-60-n9cpu4-geo-0003 | | 10.91.48.13 |
| teamcity-21520406-1789624655-60-n9cpu4-geo-0004 | | 10.91.97.222 |
| teamcity-21520406-1789624655-60-n9cpu4-geo-0005 | | 10.91.128.27 |
| teamcity-21520406-1789624655-60-n9cpu4-geo-0006 | | 10.91.48.7 |
| teamcity-21520406-1789624655-60-n9cpu4-geo-0007 | | 10.91.96.117 |
| teamcity-21520406-1789624655-60-n9cpu4-geo-0008 | | 10.91.128.12 |
| teamcity-21520406-1789624655-60-n9cpu4-geo-0009 | | 10.91.48.12 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicDRPC=true
- metamorphicLeases=expiration
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21520406/costfuzz-seed-multi-region/1789642355749/1789645070608)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21520406-1789624655-60-n9cpu4-geo&from_ts=1789642355749&to_ts=1789645070608&live=false&storage=flex_tier)

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68471

Contributor guide

Open the contributing guide

Research direction

Start with pkg/sql/opt/exec/expr/query_comparison_util.go at line 501 and inspect the artifacts and logs under /artifacts/costfuzz/seed-multi-region/run_1. Read the roachtest README and the linked investigation guide before attempting to reproduce the nine-node multi-region failure. Done means the cause of the unhandled distribute error is identified and the costfuzz run no longer fails for this case.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.