cockroachdb / cockroachdb/cockroach

kvserver: load hotspot ignored by allocator starves elastic work

Open
#173,560 7 comments 0 reactions 0 assignees View on GitHub
A-admission-control A-disaster-recovery B-runtime-assertions-enabled branch-release-26.3 C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-noreuse
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.backup-restore/small-ranges [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21473094?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21473094?buildTab=artifacts#/backup-restore/small-ranges) on release-26.3 @ [9f1d248e7525462838df0bd68f4fd71cce2de6be](https://github.com/cockroachdb/cockroach/commits/9f1d248e7525462838df0bd68f4fd71cce2de6be):

Failed with:

```
(monitor.go:267).Wait: monitor failure: retries exhausted waiting for job 1202761658170081283 to finish
test artifacts and logs in: /artifacts/backup-restore/small-ranges/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21473094-1787121479-16-n4cpu4-0001 | | 10.91.97.216 |
| teamcity-21473094-1787121479-16-n4cpu4-0002 | | 10.91.97.217 |
| teamcity-21473094-1787121479-16-n4cpu4-0003 | | 10.91.97.215 |
| teamcity-21473094-1787121479-16-n4cpu4-0004 | | 10.91.96.6 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=true
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21473094/backup-restore-small-ranges/1787123571286/1787129741723)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21473094-1787121479-16-n4cpu4&from_ts=1787123571286&to_ts=1787129741723&live=false&storage=flex_tier)

Same failure on other branches

- #172558 roachtest: backup-restore/small-ranges failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-2 T-disaster-recovery branch-master]

/cc @cockroachlabs/disaster-recovery

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*backup-restore/small-ranges.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66874

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest backup-restore/small-ranges failure at monitor.go:267, then inspect the linked artifacts, logs, Grafana dashboard, and related issue #172558. Trace the job timeout and the kvserver allocator's handling of load hotspots; done means identifying and correcting the starvation failure and getting this roachtest to complete successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.