cockroachdb / cockroachdb/cockroach

release-26.4: roachtest: perturbation/full/restart failed

Open
#175,646 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

branch-release-26.4 C-test-failure O-roachtest O-robot release-blocker T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.perturbation/full/restart failed with artifacts on release-26.4 @ b4884d371d74d172ab184bc4ef0924ccb758f3c2:

Failed with:

(assertions.go:388).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/perturbation/framework.go:1084
	            				pkg/cmd/roachtest/test_runner.go:1992
	            				src/runtime/asm_amd64.s:1771
	Error:      	FAILURE: follower-read  : throughput ratio 6508200000000.00x exceeds limit 1.2x (baseline: 6508 ops/s, measured: 0 ops/s)
	            	FAILURE: read           : throughput ratio 6515200000000.00x exceeds limit 1.2x (baseline: 6515 ops/s, measured: 0 ops/s)
	            	FAILURE: write          : throughput ratio 13090900000000.00x exceeds limit 1.2x (baseline: 13091 ops/s, measured: 0 ops/s)
	            	FAILURE: follower-read  : throughput ratio 1.40x exceeds limit 1.2x (baseline: 6508 ops/s, measured: 4640 ops/s)
	            	FAILURE: read           : throughput ratio 1.39x exceeds limit 1.2x (baseline: 6515 ops/s, measured: 4695 ops/s)
	            	FAILURE: write          : throughput ratio 1.39x exceeds limit 1.2x (baseline: 13091 ops/s, measured: 9397 ops/s)
	Test:       	perturbation/full/restart
(require.go:522).Fail: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/restart/run_1

Cluster Node to Ip Mapping:

Node Public IP Private IP
teamcity-21523842-1789803587-06-n13cpu16sm-0001 10.91.101.113
teamcity-21523842-1789803587-06-n13cpu16sm-0002 10.91.101.101
teamcity-21523842-1789803587-06-n13cpu16sm-0003 10.91.101.136
teamcity-21523842-1789803587-06-n13cpu16sm-0004 10.91.101.137
teamcity-21523842-1789803587-06-n13cpu16sm-0005 10.91.101.115
teamcity-21523842-1789803587-06-n13cpu16sm-0006 10.91.101.119
teamcity-21523842-1789803587-06-n13cpu16sm-0007 10.91.101.114
teamcity-21523842-1789803587-06-n13cpu16sm-0008 10.91.101.100
teamcity-21523842-1789803587-06-n13cpu16sm-0009 10.91.101.109
teamcity-21523842-1789803587-06-n13cpu16sm-0010 10.91.101.132
teamcity-21523842-1789803587-06-n13cpu16sm-0011 10.91.101.71
teamcity-21523842-1789803587-06-n13cpu16sm-0012 10.91.100.92
teamcity-21523842-1789803587-06-n13cpu16sm-0013 10.91.101.73

Parameters:

  • acMode=defaultOption
  • arch=amd64
  • blockSize=4096
  • cloud=gce
  • concurrency=500
  • coverageBuild=false
  • cpu=16
  • diskBandwidthLimit=0
  • diskCount=2
  • disks=2
  • encrypted=false
  • fillDuration=10m0s
  • followerReadPercent=50
  • fs=ext4
  • leaseType=leader
  • localSSD=true
  • mem=standard
  • numNodes=12
  • numWorkloadNodes=1
  • perturbationDuration=10m0s
  • ratioOfMax=0.3
  • readPercent=50
  • runtimeAssertionsBuild=false
  • seed=0
  • splits=10000
  • validationDuration=5m0s
  • vcpu=16
  • workloadConnInit=
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

/cc @cockroachlabs/kv-triage

Improve this report!

Jira issue: CRDB-68546

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the failure at pkg/cmd/roachtest/tests/perturbation/framework.go:1084 and the caller in pkg/cmd/roachtest/test_runner.go:1992. Inspect the run_1 artifacts and linked Grafana and Datadog data to determine why throughput falls to zero or below the 1.2x limit during perturbation/full/restart. Done means the failure cause is addressed and this roachtest no longer reports the shown throughput-ratio failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.