cockroachdb / cockroachdb/cockroach

sql: adversarial workload for memory pressure

Open
#101,453 1 comment 0 reactions 0 assignees View on GitHub
A-testing C-enhancement O-support P-3 quality-friday T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

We should have a workload that collects adversarial queries. For example, customers have managed, over the years, to bring CRDB down via OOMs with various queries on their datasets. Often, this involved lookup joins etc on columns containing large blobs.

We advise customers against large values, but it's not always heded and we should be resilient against it, in particular in a multi-tenant world.

These scenarios are not easy to debug, which adds insult to injury. We pay a price for this gap during support escalations.

But it's actually not trivial to recreate them (I've tried) unless you're a versed SQL Queries/Optimizer engineer or have seen the actual schema/query in the field.

**Describe the solution you'd like**

Have a roachtest that cycles through a number of adversarial scenarios, i.e. sets up a dataset and runs a number of "bad" queries on it.

Whenever we discover a new "category" of bad query, add it.

Track, per query, whether the effect on the cluster is "tolerable" (for example, watching max go allocated memory or something like that).

Initially, when a query is added as bad, it is allowed to wreck the cluster, i.e. we just restart it and continue with the next query.

As our OOM resilience improves and existing queries get "tamed", the roachtest is updated to require this query to leave the cluster healthy.

Any proposed observability improvements or guardrails would be prototyped against this test.

**Describe alternatives you've considered**

**Additional context**

Jira issue: CRDB-26957

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.