cockroachdb / cockroachdb/cockroach

kv: nodes can OOM with low memory configurations

Open
#137,666 4 comments 0 reactions 0 assignees View on GitHub
A-testing branch-master C-bug O-roachtest P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Nodes can OOM with low memory configurations for metamorphic tests. Specifically the run below with seed 797788059808040562 for the partition test hits an OOM during recovery.

This may be protected once we have apply_to_all since that protects an unbounded amount of raft traffic.

I will retest in that mode.

roachtest.perturbation/metamorphic/partition [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_PerturbationNightlyGceBazel/18228565?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_PerturbationNightlyGceBazel/18228565?buildTab=artifacts#/perturbation/metamorphic/partition) on master @ [51693691ed763f700dd06fa2d001cce1ffd42203](https://github.com/cockroachdb/cockroach/commits/51693691ed763f700dd06fa2d001cce1ffd42203):

```
(assertions.go:363).Fail:
Error Trace: pkg/cmd/roachtest/roachtestutil/validation_check.go:158
pkg/testutils/soon.go:71
pkg/util/retry/retry.go:213
pkg/testutils/soon.go:77
pkg/testutils/soon.go:56
pkg/cmd/roachtest/roachtestutil/validation_check.go:142
pkg/cmd/roachtest/tests/perturbation/framework.go:681
pkg/cmd/roachtest/test_runner.go:1314
src/runtime/asm_amd64.s:1695
Error: Received unexpected error:
dial tcp 34.139.20.73:26257: connect: connection refused
Test: perturbation/metamorphic/partition
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/perturbation/metamorphic/partition/run_1
```

Parameters:
- acMode=fullNormalElasticRepl
- arch=amd64
- blockSize=4096
- cloud=gce
- coverageBuild=false
- cpu=4
- diskBandwidthLimit=0
- disks=1
- encrypted=false
- fillDuration=10m0s
- fs=ext4
- leaseType=epoch
- localSSD=true
- mem=low
- numNodes=30
- numWorkloadNodes=2
- perturbationDuration=30m0s
- ratioOfMax=0.5
- runtimeAssertionsBuild=false
- seed=797788059808040562
- splits=100
- ssd=1
- validationDuration=5m0s
- vcpu=4
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-18228565/perturbation-metamorphic-partition/1734505340937/1734508750706)

/cc @cockroachdb/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*perturbation/metamorphic/partition.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-45686

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.