cockroachdb / cockroachdb/cockroach

create and drop database in rapid succession in testing cause GC pile up and 100% CPU usage

Open
#173,353 1 comment 0 reactions 0 assignees View on GitHub
C-enhancement O-community X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

In testing, there a pattern where you can create a database to test a feature and drop it when done, but in cockroachdb, the engine for create/drop database is not fast enough - gc job pile up and cause 100% cpu usage, this is what I'm tunning:

```
ALTER RANGE default CONFIGURE ZONE USING gc.ttlseconds = 60;
SET CLUSTER SETTING jobs.registry.interval.gc = '2m';
SET CLUSTER SETTING jobs.retention_time = '10m';
SET CLUSTER SETTING kv.mvcc_gc.queue_interval = '0s';
SET CLUSTER SETTING kv.range_merge.queue_interval = '0s';
SET CLUSTER SETTING sql.gc_job.wait_for_gc.interval = '30s';
SET CLUSTER SETTING sql.schema.approx_max_object_count = 100000;
SET CLUSTER SETTING sql.stats.automatic_collection.enabled = false;
```

Even with these setting, it seem like GC is still very slow compare to test, just in 30s, application testing can generate a few thousands of GC job, jobs pile up much faster than it drain and application testing stand to a halt or worse - failling because CPU starvation.

For now, I combat this by using cockroachdb on tmpfs, it will drop all data and create new cluster every time I restart crdb service - this is to side step the problem - but if there are too many tests, not even this can help..

**Describe the solution you'd like**
Super duper fast drop database - with almost no cpu usage, currently this is only for testing but if you can pull this off, it will create new usage pattern for crdb.

Jira issue: CRDB-66669

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the rapid create/drop testing workload in CockroachDB with the listed GC and job settings, then observe GC job backlog and CPU usage. Done means database creation and removal no longer cause jobs to pile up or testing to halt; the issue does not name source files or tests to inspect.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.