cockroachdb / cockroachdb/cockroach

changefeedccl: changefeeds can use all your memory

Open
#156,005 1 comment 0 reactions 0 assignees View on GitHub
A-cdc branch-master C-bug T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Each changefeed requests an amount of memory from its parent monitor with a lower and upper limit. If you create enough changefeeds this can exhaust the budget of its parent, and its parent, up to the root. This can impact the cluster's ability to serve foreground traffic and keep itself healthy.

Example reproduction:

- start cockroach with --max-sql-memory=1GiB to make things easier
- in another terminal, run the `cdc-tools/miles/webhook-sink` program (we'll use the `/blackhole` endpoint to increase buffering per feed)
- in a sql shell:
```
> create table t (a string);
> insert into t (a) select 'hello-' || g::string from generate_series(1, 50000) as g;

-- keep making identical feeds until the logs start to freak out. 55 did it for me
> CREATE CHANGEFEED FOR t INTO 'webhook-https://localhost:9090/blackhole?insecure_tls_skip_verify=true';
...etc...

> select count(*) from t;
ERROR: root: memory budget exceeded: 10240 bytes requested, 1073732034 currently allocated, 1073741824 bytes in budget
```

And if you check the db console you can see `bulk` is using all of `root`'s memory:

Image

The solution is to fix the memory monitor hierarchy to 1) actually use the changefeed monitor, and 2) make sure it's capped and won't eat all its parent's memory.

Jira issue: CRDB-55738

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.