cockroachdb / cockroachdb/cockroach
create and drop database in rapid succession in testing cause GC pile up and 100% CPU usage
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
In testing, there a pattern where you can create a database to test a feature and drop it when done, but in cockroachdb, the engine for create/drop database is not fast enough - gc job pile up and cause 100% cpu usage, this is what I'm tunning:
```
ALTER RANGE default CONFIGURE ZONE USING gc.ttlseconds = 60;
SET CLUSTER SETTING jobs.registry.interval.gc = '2m';
SET CLUSTER SETTING jobs.retention_time = '10m';
SET CLUSTER SETTING kv.mvcc_gc.queue_interval = '0s';
SET CLUSTER SETTING kv.range_merge.queue_interval = '0s';
SET CLUSTER SETTING sql.gc_job.wait_for_gc.interval = '30s';
SET CLUSTER SETTING sql.schema.approx_max_object_count = 100000;
SET CLUSTER SETTING sql.stats.automatic_collection.enabled = false;
```
Even with these setting, it seem like GC is still very slow compare to test, just in 30s, application testing can generate a few thousands of GC job, jobs pile up much faster than it drain and application testing stand to a halt or worse - failling because CPU starvation.
For now, I combat this by using cockroachdb on tmpfs, it will drop all data and create new cluster every time I restart crdb service - this is to side step the problem - but if there are too many tests, not even this can help..
**Describe the solution you'd like**
Super duper fast drop database - with almost no cpu usage, currently this is only for testing but if you can pull this off, it will create new usage pattern for crdb.
Jira issue: CRDB-66669
Contributor guide
Research direction
Start by reproducing the rapid create/drop testing workload in CockroachDB with the listed GC and job settings, then observe GC job backlog and CPU usage. Done means database creation and removal no longer cause jobs to pile up or testing to halt; the issue does not name source files or tests to inspect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100