cockroachdb / cockroachdb/cockroach

release-26.4: roachtest: replicate/wide failed

Open
#175,656 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

branch-release-26.4 C-test-failure O-roachtest O-robot release-blocker T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.replicate/wide failed with artifacts on release-26.4 @ b4884d371d74d172ab184bc4ef0924ccb758f3c2:

Failed with:

(assertions.go:388).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/allocator.go:458
	            				pkg/cmd/roachtest/tests/allocator.go:464
	            				pkg/cmd/roachtest/test_runner.go:1992
	            				src/runtime/asm_amd64.s:1771
	Error:      	Received unexpected error:
	            	roachtest node 9 is not a CRDB node
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.validateCockroachPhysicalNode
	            	  | 	pkg/cmd/roachtest/cluster_node_identities.go:112
	            	  | main.cockroachNode
	            	  | 	pkg/cmd/roachtest/cluster_node_identities.go:198
	            	  | main.(*roachprodCluster).CockroachNode
	            	  | 	pkg/cmd/roachtest/cluster_node_identities.go:229
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runWideReplication.func8
	            	  | 	pkg/cmd/roachtest/tests/allocator.go:457
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runWideReplication
	            	  | 	pkg/cmd/roachtest/tests/allocator.go:464
	            	  | main.(*testRunner).runTest.func2
	            	  | 	pkg/cmd/roachtest/test_runner.go:1992
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1771
	            	Wraps: (2) roachtest node 9 is not a CRDB node
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	replicate/wide
(require.go:1376).NoError: FailNow called
test artifacts and logs in: /artifacts/replicate/wide/run_1

Cluster Node to Ip Mapping:

Node Public IP Private IP
teamcity-21523842-1789803587-55-n9cpu1-0001 10.91.98.75
teamcity-21523842-1789803587-55-n9cpu1-0002 10.91.99.235
teamcity-21523842-1789803587-55-n9cpu1-0003 10.91.99.36
teamcity-21523842-1789803587-55-n9cpu1-0004 10.91.98.98
teamcity-21523842-1789803587-55-n9cpu1-0005 10.91.98.76
teamcity-21523842-1789803587-55-n9cpu1-0006 10.91.97.46
teamcity-21523842-1789803587-55-n9cpu1-0007 10.91.98.62
teamcity-21523842-1789803587-55-n9cpu1-0008 10.91.98.110
teamcity-21523842-1789803587-55-n9cpu1-0009 10.91.98.24

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=1
  • diskCount=0
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicLeases=default
  • runtimeAssertionsBuild=false
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

Same failure on other branches

  • #175650 roachtest: replicate/wide failed [C-test-failure O-roachtest O-robot T-kv branch-master release-blocker]

/cc @cockroachlabs/kv-triage

Improve this report!

Jira issue: CRDB-68556

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pkg/cmd/roachtest/tests/allocator.go at lines 457-464, then trace node validation through cluster_node_identities.go at lines 112, 198, and 229. Review the linked TeamCity artifacts and logs for replicate/wide and compare the related failure in #175650. Done means replicate/wide no longer reports node 9 as a non-CRDB node on the affected branches.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.