cockroachdb / cockroachdb/cockroach

basalt-26.3: roachtest: db-console/mixed-version-cypress failed

Open
#175,393 2 comments 0 reactions 0 assignees View on GitHub
branch-basalt-26.3 C-test-failure O-roachtest O-robot release-blocker T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.db-console/mixed-version-cypress [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21514441?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21514441?buildTab=artifacts#/db-console/mixed-version-cypress) on basalt-26.3 @ [4fda4910b2015319b2f775e42d2f2fb15130d19a](https://github.com/cockroachdb/cockroach/commits/4fda4910b2015319b2f775e42d2f2fb15130d19a):

Failed with:

```
(assertions.go:388).Fail:
Error Trace: pkg/cmd/roachtest/tests/db_console_cypress.go:109
pkg/cmd/roachtest/tests/db_console_cypress.go:311
pkg/cmd/roachtest/roachtestutil/mixedversion/steps.go:468
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:268
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:819
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:267
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:242
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:224
pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:217
pkg/util/ctxgroup/ctxgroup.go:200
external/org_golang_x_sync/errgroup/errgroup.go:93
src/runtime/asm_amd64.s:1771
Error: Received unexpected error:
full command output in run_070404.698271215_n5_cd-tmpdbconsolecypre.log: COMMAND_PROBLEM: exit status 1
(1) attached stack trace
-- stack trace:
| main.(*roachprodCluster).RunE
| pkg/cmd/roachtest/cluster.go:4394
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*dbConsoleCypressTest).RunTest
| pkg/cmd/roachtest/tests/db_console_cypress.go:109
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runDbConsoleCypressMixedVersions.func2
| pkg/cmd/roachtest/tests/db_console_cypress.go:311
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.runHookStep.Run
| pkg/cmd/roachtest/roachtestutil/mixedversion/steps.go:468
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runSingleStep.func2
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:268
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.panicAsError
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:819
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runSingleStep
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:267
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runStep
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:242
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runStep
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:224
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runStep.func1
| pkg/cmd/roachtest/roachtestutil/mixedversion/runner.go:217
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*testRunner).runStep.Group.GoCtx.func2
| pkg/util/ctxgroup/ctxgroup.go:200
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_amd64.s:1771
Wraps: (2) full command output in run_070404.698271215_n5_cd-tmpdbconsolecypre.log
Wraps: (3) Node 5. Command with error:
| ``````
| cd /tmp/dbconsole-cypress && NO_COLOR=1 npx cypress run --e2e --config baseUrl=https://18.220.104.181:26258,screenshotsFolder=/tmp/dbconsole-cypress/artifacts,videosFolder=/tmp/dbconsole-cypress/artifacts --spec "cypress/e2e/health-check/*.ts"
| ``````
| ... �────────────────────────────────────────────────────────────────────────────────────┤
| │ ✔ unauthenticated.cy.ts 00:05 1 1 - - - │
| └────────────────────────────────────────────────────────────────────────────────────────────────┘
| ✖ 1 of 2 failed (50%) 00:42 2 1 1 - -
|
| stderr:DevTools listening on ws://127.0.0.1:34843/devtools/browser/c6039725-4735-438b-beb0-66c846c88f55
| Missing baseUrl in compilerOptions. tsconfig-paths will be skipped
| tput: No value for $TERM and no -T specified
| tput: No value for $TERM and no -T specified
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *hintdetail.withDetail (4) errors.Cmd (5) *exec.ExitError
Test: db-console/mixed-version-cypress
(db_console_cypress.go:115).RunTest: full command output in run_070404.698271215_n5_cd-tmpdbconsolecypre.log: COMMAND_PROBLEM: exit status 1
(mixedversion.go:926).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/db-console/mixed-version-cypress/cpu_arch=arm64/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21514441-1789279054-42-n5cpu4-0001 | 18.191.141.190 | 10.12.30.24 |
| teamcity-21514441-1789279054-42-n5cpu4-0002 | 18.220.104.181 | 10.12.28.5 |
| teamcity-21514441-1789279054-42-n5cpu4-0003 | 18.219.206.238 | 10.12.24.24 |
| teamcity-21514441-1789279054-42-n5cpu4-0004 | 13.58.62.188 | 10.12.21.160 |
| teamcity-21514441-1789279054-42-n5cpu4-0005 | 18.222.218.122 | 10.12.23.93 |

Parameters:
- arch=arm64
- cloud=aws
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- mvtDeploymentMode=system-only
- mvtVersions=v26.2.3 → v26.3.2-dev-4fda4910b2015319b2f775e42d2f2fb15130d19a (basalt-26.3)
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21514441/db-console-mixed-version-cypress/1789282826169/1789283253769)

_Logs were not uploaded to Datadog_

Same failure on other branches

- #175390 release-26.3: roachtest: db-console/mixed-version-cypress failed [C-test-failure O-roachtest O-robot T-observability branch-release-26.3 release-blocker]
- #175387 roachtest: db-console/mixed-version-cypress failed [C-test-failure O-roachtest O-robot T-observability branch-release-25.4 release-blocker]
- #175386 roachtest: db-console/mixed-version-cypress failed [C-test-failure O-roachtest O-robot T-observability branch-release-25.2 release-blocker]
- #174895 roachtest: db-console/mixed-version-cypress failed [C-test-failure O-roachtest O-robot T-observability branch-master]

/cc @cockroachlabs/obs-prs

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*db-console/mixed-version-cypress.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68348

Contributor guide

Open the contributing guide

Research direction

Inspect pkg/cmd/roachtest/tests/db_console_cypress.go at the reported lines, then review run_070404.698271215_n5_cd-tmpdbconsolecypre.log and the attached artifacts. Reproduce the npx cypress run command for cypress/e2e/health-check/*.ts and determine why one health-check fails in the mixed-version test. Done means the test passes reliably across the affected branches.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cypress, go, typescript
Domain
cloud, databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.