cockroachdb / cockroachdb/cockroach
kv: nodes can OOM with low memory configurations
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Nodes can OOM with low memory configurations for metamorphic tests. Specifically the run below with seed 797788059808040562 for the partition test hits an OOM during recovery.
This may be protected once we have apply_to_all since that protects an unbounded amount of raft traffic.
I will retest in that mode.
roachtest.perturbation/metamorphic/partition [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_PerturbationNightlyGceBazel/18228565?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_PerturbationNightlyGceBazel/18228565?buildTab=artifacts#/perturbation/metamorphic/partition) on master @ [51693691ed763f700dd06fa2d001cce1ffd42203](https://github.com/cockroachdb/cockroach/commits/51693691ed763f700dd06fa2d001cce1ffd42203):
```
(assertions.go:363).Fail:
Error Trace: pkg/cmd/roachtest/roachtestutil/validation_check.go:158
pkg/testutils/soon.go:71
pkg/util/retry/retry.go:213
pkg/testutils/soon.go:77
pkg/testutils/soon.go:56
pkg/cmd/roachtest/roachtestutil/validation_check.go:142
pkg/cmd/roachtest/tests/perturbation/framework.go:681
pkg/cmd/roachtest/test_runner.go:1314
src/runtime/asm_amd64.s:1695
Error: Received unexpected error:
dial tcp 34.139.20.73:26257: connect: connection refused
Test: perturbation/metamorphic/partition
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/perturbation/metamorphic/partition/run_1
```
Parameters:
- acMode=fullNormalElasticRepl
- arch=amd64
- blockSize=4096
- cloud=gce
- coverageBuild=false
- cpu=4
- diskBandwidthLimit=0
- disks=1
- encrypted=false
- fillDuration=10m0s
- fs=ext4
- leaseType=epoch
- localSSD=true
- mem=low
- numNodes=30
- numWorkloadNodes=2
- perturbationDuration=30m0s
- ratioOfMax=0.5
- runtimeAssertionsBuild=false
- seed=797788059808040562
- splits=100
- ssd=1
- validationDuration=5m0s
- vcpu=4
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-18228565/perturbation-metamorphic-partition/1734505340937/1734508750706)
/cc @cockroachdb/kv-triage
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*perturbation/metamorphic/partition.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-45686
Contributor guide
Assessment
This issue has not been assessed yet.