cockroachdb / cockroachdb/cockroach

cli: sql memory limit can exceed go memory limit

Open
#153,041 1 comment 0 reactions 0 assignees View on GitHub
branch-master
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

CockroachDB has two flags to control the memory usage of a process.

```
--max-go-memory
Soft memory limit set on the Go runtime (which is also configurable via the
GOMEMLIMIT environment variable, but --max-go-memory has higher precedence
if both are set). Notably, the pebble cache (as configured by --cache) is not
under control of the Go runtime and should not be considered when determining
this soft memory limit. Accepts numbers interpreted as bytes, size suffixes
(e.g. 1GB and 1GiB) or a percentage of physical memory (e.g. .25). If left
unspecified, defaults to 2.25x of --max-sql-memory (subject to max-go-memory
+ 1.15x --cache not exceeding 90% of available RAM). Set to 0 to disable the
soft memory limit (not recommended).

--max-sql-memory
Maximum memory capacity available to store temporary data for SQL clients,
including prepared queries and intermediate data rows during query execution.
Accepts numbers interpreted as bytes, size suffixes (e.g. 1GB and 1GiB) or a
percentage of physical memory (e.g. .25). If left unspecified, defaults to 25%
of physical memory.
```

There is a bad interaction between the `GOMEMLIMIT` environment variable and the `--max-sql-memory` flag. If both flags are configured, the database will ensure the go memory limit is greater than the SQL memory limit. But if the `GOMEMLIMIT` environment variable is set, its possible the sql-memory-limit will exceed the `GOMEMLIMIT`. This leads to a metastable configuration where the sql layer will not limit memory usage before the memory usage outgrows the memory limit.

The symptom for this configuration is a profile that is spending almost 100% of its cpu time in the GC.

## Workaround

If customers set GOMEMLIMIT, they should set an absolute --max-sql-memory such that max-sql-memory * 2.25 == GOMEMLIMIT.

## Proposed Fix

If GOMEMLIMIT is set, we should run the same validation that we would if --max-go-memory was set.

Jira issue: CRDB-54168

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.