cockroachdb / cockroachdb/cockroach

roachtest: c2c/shutdown/src/coordinator failed

Abierto
#175,507 1 comentario 0 reacciones 0 asignados Ver en GitHub
A-disaster-recovery B-runtime-assertions-enabled branch-master C-test-failure O-roachtest O-robot release-blocker T-disaster-recovery
Lenguaje dominante
Go
Estrellas
32.5k
Forks
4.1k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.c2c/shutdown/src/coordinator [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21516773?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21516773?buildTab=artifacts#/c2c/shutdown/src/coordinator) on master @ [b112dc518c826da8b4b577934d4ab5c0c24da045](https://github.com/cockroachdb/cockroach/commits/b112dc518c826da8b4b577934d4ab5c0c24da045):

Failed with:

```
(cluster_to_cluster.go:2128).func1: shutdown execution failed: could not restart node :3: failed to wait for SQL to be ready: open /tmp/roachtest-certs2416886761/certs/ca.crt: no such file or directory
(assertions.go:388).Fail:
Error Trace: pkg/cmd/roachtest/tests/cluster_to_cluster.go:1137
pkg/cmd/roachtest/tests/cluster_to_cluster.go:1408
pkg/cmd/roachtest/tests/cluster_to_cluster.go:2079
pkg/cmd/roachtest/monitor.go:295
external/org_golang_x_sync/errgroup/errgroup.go:93
src/runtime/asm_amd64.s:1771
Error: Received unexpected error:
failed stripped fingerprint: failed getting dst fingerprint: could not get database names: context canceled
(1) secondary error attachment
| already failed
| (1) attached stack trace
| -- stack trace:
| | main.(*monitorImpl).WaitE
| | pkg/cmd/roachtest/monitor.go:314
| | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).onFingerprintMismatch
| | pkg/cmd/roachtest/tests/cluster_to_cluster.go:1136
| | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
| | pkg/cmd/roachtest/tests/cluster_to_cluster.go:1408
| | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterReplicationResilience.func1.3
| | pkg/cmd/roachtest/tests/cluster_to_cluster.go:2079
| | main.(*monitorImpl).Go.func1
| | pkg/cmd/roachtest/monitor.go:295
| | golang.org/x/sync/errgroup.(*Group).Go.func1
| | external/org_golang_x_sync/errgroup/errgroup.go:93
| | runtime.goexit
| | src/runtime/asm_amd64.s:1771
| Wraps: (2) already failed
| Error types: (1) *withstack.withStack (2) *errutil.leafError
Wraps: (2) failed stripped fingerprint
Wraps: (3) failed getting dst fingerprint
Wraps: (4) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/testutils/fingerprintutils.FingerprintAllDatabases
| pkg/testutils/fingerprintutils/fingerprint.go:178
| github.com/cockroachdb/cockroach/pkg/crosscluster/replicationutils.fingerprintClustersByTable.func2
| pkg/crosscluster/replicationutils/utils.go:284
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_amd64.s:1771
Wraps: (5) could not get database names
Wraps: (6) context canceled
Error types: (1) *secondary.withSecondaryError (2) *fmt.wrapError (3) *fmt.wrapError (4) *withstack.withStack (5) *errutil.withPrefix (6) *errors.errorString
Test: c2c/shutdown/src/coordinator
(require.go:1376).NoError: FailNow called
test artifacts and logs in: /artifacts/c2c/shutdown/src/coordinator/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21516773-1789451745-32-n9cpu8-0001 | | 10.91.99.192 |
| teamcity-21516773-1789451745-32-n9cpu8-0002 | | 10.91.99.203 |
| teamcity-21516773-1789451745-32-n9cpu8-0003 | | 10.91.99.158 |
| teamcity-21516773-1789451745-32-n9cpu8-0004 | | 10.91.98.179 |
| teamcity-21516773-1789451745-32-n9cpu8-0005 | | 10.91.99.176 |
| teamcity-21516773-1789451745-32-n9cpu8-0006 | | 10.91.99.190 |
| teamcity-21516773-1789451745-32-n9cpu8-0007 | | 10.91.98.243 |
| teamcity-21516773-1789451745-32-n9cpu8-0008 | | 10.91.99.204 |
| teamcity-21516773-1789451745-32-n9cpu8-0009 | | 10.91.99.162 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=8
- diskCount=0
- encrypted=false
- metamorphicLeases=expiration
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21516773/c2c-shutdown-src-coordinator/1789465599141/1789466448143)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21516773-1789451745-32-n9cpu8&from_ts=1789465599141&to_ts=1789466448143&live=false&storage=flex_tier)

/cc @cockroachlabs/disaster-recovery

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68423

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.