cockroachdb / cockroachdb/cockroach
changefeedccl: changefeeds can use all your memory
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Each changefeed requests an amount of memory from its parent monitor with a lower and upper limit. If you create enough changefeeds this can exhaust the budget of its parent, and its parent, up to the root. This can impact the cluster's ability to serve foreground traffic and keep itself healthy.
Example reproduction:
- start cockroach with --max-sql-memory=1GiB to make things easier
- in another terminal, run the `cdc-tools/miles/webhook-sink` program (we'll use the `/blackhole` endpoint to increase buffering per feed)
- in a sql shell:
```
> create table t (a string);
> insert into t (a) select 'hello-' || g::string from generate_series(1, 50000) as g;
-- keep making identical feeds until the logs start to freak out. 55 did it for me
> CREATE CHANGEFEED FOR t INTO 'webhook-https://localhost:9090/blackhole?insecure_tls_skip_verify=true';
...etc...
> select count(*) from t;
ERROR: root: memory budget exceeded: 10240 bytes requested, 1073732034 currently allocated, 1073741824 bytes in budget
```
And if you check the db console you can see `bulk` is using all of `root`'s memory:
The solution is to fix the memory monitor hierarchy to 1) actually use the changefeed monitor, and 2) make sure it's capped and won't eat all its parent's memory.
Jira issue: CRDB-55738
Contributor guide
Assessment
This issue has not been assessed yet.