cockroachdb / cockroachdb/cockroach

roachtest: c2c/shutdown/src/coordinator failed

Đang mở
#175,507 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
A-disaster-recovery B-runtime-assertions-enabled branch-master C-test-failure O-roachtest O-robot release-blocker T-disaster-recovery
Ngôn ngữ chính
Go
Star
32.5k
Fork
4.1k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.c2c/shutdown/src/coordinator [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21516773?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21516773?buildTab=artifacts#/c2c/shutdown/src/coordinator) on master @ [b112dc518c826da8b4b577934d4ab5c0c24da045](https://github.com/cockroachdb/cockroach/commits/b112dc518c826da8b4b577934d4ab5c0c24da045):

Failed with:

```
(cluster_to_cluster.go:2128).func1: shutdown execution failed: could not restart node :3: failed to wait for SQL to be ready: open /tmp/roachtest-certs2416886761/certs/ca.crt: no such file or directory
(assertions.go:388).Fail:
Error Trace: pkg/cmd/roachtest/tests/cluster_to_cluster.go:1137
pkg/cmd/roachtest/tests/cluster_to_cluster.go:1408
pkg/cmd/roachtest/tests/cluster_to_cluster.go:2079
pkg/cmd/roachtest/monitor.go:295
external/org_golang_x_sync/errgroup/errgroup.go:93
src/runtime/asm_amd64.s:1771
Error: Received unexpected error:
failed stripped fingerprint: failed getting dst fingerprint: could not get database names: context canceled
(1) secondary error attachment
| already failed
| (1) attached stack trace
| -- stack trace:
| | main.(*monitorImpl).WaitE
| | pkg/cmd/roachtest/monitor.go:314
| | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).onFingerprintMismatch
| | pkg/cmd/roachtest/tests/cluster_to_cluster.go:1136
| | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
| | pkg/cmd/roachtest/tests/cluster_to_cluster.go:1408
| | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterReplicationResilience.func1.3
| | pkg/cmd/roachtest/tests/cluster_to_cluster.go:2079
| | main.(*monitorImpl).Go.func1
| | pkg/cmd/roachtest/monitor.go:295
| | golang.org/x/sync/errgroup.(*Group).Go.func1
| | external/org_golang_x_sync/errgroup/errgroup.go:93
| | runtime.goexit
| | src/runtime/asm_amd64.s:1771
| Wraps: (2) already failed
| Error types: (1) *withstack.withStack (2) *errutil.leafError
Wraps: (2) failed stripped fingerprint
Wraps: (3) failed getting dst fingerprint
Wraps: (4) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/testutils/fingerprintutils.FingerprintAllDatabases
| pkg/testutils/fingerprintutils/fingerprint.go:178
| github.com/cockroachdb/cockroach/pkg/crosscluster/replicationutils.fingerprintClustersByTable.func2
| pkg/crosscluster/replicationutils/utils.go:284
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_amd64.s:1771
Wraps: (5) could not get database names
Wraps: (6) context canceled
Error types: (1) *secondary.withSecondaryError (2) *fmt.wrapError (3) *fmt.wrapError (4) *withstack.withStack (5) *errutil.withPrefix (6) *errors.errorString
Test: c2c/shutdown/src/coordinator
(require.go:1376).NoError: FailNow called
test artifacts and logs in: /artifacts/c2c/shutdown/src/coordinator/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21516773-1789451745-32-n9cpu8-0001 | | 10.91.99.192 |
| teamcity-21516773-1789451745-32-n9cpu8-0002 | | 10.91.99.203 |
| teamcity-21516773-1789451745-32-n9cpu8-0003 | | 10.91.99.158 |
| teamcity-21516773-1789451745-32-n9cpu8-0004 | | 10.91.98.179 |
| teamcity-21516773-1789451745-32-n9cpu8-0005 | | 10.91.99.176 |
| teamcity-21516773-1789451745-32-n9cpu8-0006 | | 10.91.99.190 |
| teamcity-21516773-1789451745-32-n9cpu8-0007 | | 10.91.98.243 |
| teamcity-21516773-1789451745-32-n9cpu8-0008 | | 10.91.99.204 |
| teamcity-21516773-1789451745-32-n9cpu8-0009 | | 10.91.99.162 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=8
- diskCount=0
- encrypted=false
- metamorphicLeases=expiration
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21516773/c2c-shutdown-src-coordinator/1789465599141/1789466448143)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21516773-1789451745-32-n9cpu8&from_ts=1789465599141&to_ts=1789466448143&live=false&storage=flex_tier)

/cc @cockroachlabs/disaster-recovery

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68423

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.