cockroachdb / cockroachdb/cockroach

multitenant: Throttled queries do not make progress once distributed token bucket is refilled

Open
#101,813 1 comment 0 reactions 0 assignees View on GitHub
A-multitenancy C-bug
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

A SQL pod/process that becomes throttled remains throttled even once plenty of tokens are filled in the global/distributed token bucket (`system.tenant_usage`).

**To Reproduce**

Set up a multi-tenant CRDB cluster, create a secondary tenant (in this example tenant id=2), and user/password for tenant.
On the system tenant, set the resource limits for tenant-2 such that it has a small amount of RUs and no refill rate

```
// for tenant with id=2, set 10000 RUs, 0 refill rate
select crdb_internal.update_tenant_resource_limits(2, 10000, 0, 0, now(), 0);
```

Initialize a schema heavy workload (tpcc). This consumes more than 10000 RUs and the sql pod/process will get throttled (the loading of schemas will hang)

```
alyshanjahani@crlMBP-C02ZP0C2MD6TMTIy ~ % cockroach workload init tpcc "postgresql://:@localhost:26257/tpcc?sslmode=require"
I230418 17:14:20.448375 1 workload/workloadsql/dataload.go:146 [-] 1 imported warehouse (0s, 1 rows)
I230418 17:14:20.526387 1 workload/workloadsql/dataload.go:146 [-] 2 imported district (0s, 10 rows)
// at this point the command is hanging
```

On the system tenant, you'll notice that the token bucket for the tenant is in debt (negative value)
```
root@localhost:26257/defaultdb> select ru_current from system.tenant_usage where instance_id=0 and tenant_id=2;
-[ RECORD 1 ]
ru_current | -37985.90773410235
```

Set the token bucket to be full again with a large value like 100M RUs
```
select crdb_internal.update_tenant_resource_limits(2, 100000000, 0, 0, now(), 0);
```

The command is still hanging, and other connections and queries fail/hang as well.
However, if we spin up a new SQL pod for the tenant, connections and queries on that pod will succeed.

**Expected behavior**

The queries on the SQL process should be able to make progress again once tokens have been filled in the global token bucket for that tenant.

It seems like the SQL process is not reaching out to KV / the global token bucket to refill its local token bucket with the newly added tokens as described [here](https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20210604_distributed_token_bucket.md#kv-connector-api)

**Environment:**
- CockroachDB version: 22.2.7
- tested in a K8s environment (on a K3D cluster as well as a GKE cluster)

Jira issue: CRDB-27133

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.