cockroachdb / cockroachdb/cockroach

release-26.4: roachtest: replicagc-changed-peers/restart=true failed

Open
#175,690 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

B-runtime-assertions-enabled branch-release-26.4 C-test-failure O-roachtest O-robot release-blocker T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.replicagc-changed-peers/restart=true failed with artifacts on release-26.4 @ b4884d371d74d172ab184bc4ef0924ccb758f3c2:

Failed with:

(replicagc.go:337).waitForZeroReplicasOnN3: pq: could not determine data type of placeholder $1
test artifacts and logs in: /artifacts/replicagc-changed-peers/restart=true/run_1

Cluster Node to Ip Mapping:

Node Public IP Private IP
teamcity-21524579-1789881036-97-n6cpu4-0001 10.91.99.45
teamcity-21524579-1789881036-97-n6cpu4-0002 10.91.97.206
teamcity-21524579-1789881036-97-n6cpu4-0003 10.91.97.239
teamcity-21524579-1789881036-97-n6cpu4-0004 10.91.99.67
teamcity-21524579-1789881036-97-n6cpu4-0005 10.91.97.118
teamcity-21524579-1789881036-97-n6cpu4-0006 10.91.99.62

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • diskCount=0
  • encrypted=false
  • fs=ext4
  • localSSD=true
  • metamorphicLeases=default
  • runtimeAssertionsBuild=true
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

Same failure on other branches

  • #175670 release-26.2: roachtest: replicagc-changed-peers/restart=true failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-kv branch-release-26.2 release-blocker]
  • #175653 roachtest: replicagc-changed-peers/restart=true failed [C-test-failure O-roachtest O-robot T-kv branch-master release-blocker]

/cc @cockroachlabs/kv-triage

Improve this report!

Jira issue: CRDB-68576

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the roachtest.replicagc-changed-peers/restart=true failure and inspect pkg/cmd/roachtest/replicagc.go around line 337, waitForZeroReplicasOnN3. Review the linked TeamCity artifacts and logs for the placeholder error, then reproduce the test if possible. Done means the underlying failure is explained and the roachtest no longer fails for this scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.