cockroachdb / cockroachdb/cockroach

roachtest: kv50/rangelookups/relocate/nodes=8 failed

Open
#175,610 4 comments 0 reactions 1 assignee Assigned to @iskettaneh View on GitHub
A-kv branch-master C-bug C-test-failure O-roachtest O-robot P-2 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.kv50/rangelookups/relocate/nodes=8 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21521840?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21521840?buildTab=artifacts#/kv50/rangelookups/relocate/nodes=8) on master @ [fc4bab5cd872ac0384f46264ff2cf147f26819cd](https://github.com/cockroachdb/cockroach/commits/fc4bab5cd872ac0384f46264ff2cf147f26819cd):

Failed with:

```
(monitor.go:329).Wait: monitor failure: pq: internal error: range r160:/Table/106/1/12{80767945177636192-99196260935587792} [(n4,s4):11, (n7,s7):12, (n8,s8):8, (n5,s5):13LEARNER, next=14, gen=73, sticky=9223372036.854775807,2147483647] was either in a joint configuration or had learner replicas: (n4,s4):11,(n7,s7):12,(n8,s8):8,(n5,s5):13LEARNER
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21521840-1789709901-88-n9cpu8-0001 | 20.127.208.77 | 10.1.0.15 |
| teamcity-21521840-1789709901-88-n9cpu8-0002 | 52.146.90.59 | 10.1.0.12 |
| teamcity-21521840-1789709901-88-n9cpu8-0003 | 20.121.138.150 | 10.1.0.126 |
| teamcity-21521840-1789709901-88-n9cpu8-0004 | 20.42.116.107 | 10.1.0.124 |
| teamcity-21521840-1789709901-88-n9cpu8-0005 | 20.127.243.23 | 10.1.0.101 |
| teamcity-21521840-1789709901-88-n9cpu8-0006 | 40.76.98.107 | 10.1.0.125 |
| teamcity-21521840-1789709901-88-n9cpu8-0007 | 20.55.83.152 | 10.1.0.149 |
| teamcity-21521840-1789709901-88-n9cpu8-0008 | 20.102.109.4 | 10.1.0.102 |
| teamcity-21521840-1789709901-88-n9cpu8-0009 | 20.127.149.140 | 10.1.0.163 |

Parameters:
- MetamorphicFilesystem=ext4
- MetamorphicVolumeType=local-ssd
- arch=amd64
- cloud=azure
- coverageBuild=false
- cpu=8
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicDRPC=true
- metamorphicLeases=epoch
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21521840/kv50-rangelookups-relocate-nodes-8/1789720477623/1789721272764)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21521840-1789709901-88-n9cpu8&from_ts=1789720477623&to_ts=1789721272764&live=false&storage=flex_tier)

/cc @cockroachlabs/kv-triage

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68497

Contributor guide

Open the contributing guide

Research direction

Begin with the failed roachtest artifacts and logs at /artifacts/kv50/rangelookups/relocate/nodes=8/run_1, then consult the roachtest README and linked Grafana or Datadog data. Trace the kv50/rangelookups/relocate/nodes=8 failure and determine whether the learner or joint-configuration error is reproducible. Done means the failure's cause is identified and an appropriate regression fix or test change is validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.