cockroachdb / cockroachdb/cockroach

mixedversion,roachtest: integrate SQL proxy with framework

Open
#164,366 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-foundations A-testing C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Now that we have [SQL Proxy support in roachtests](https://github.com/cockroachdb/cockroach/pull/161536), we should integrate it with the mixed version framework, who is the main user of separate process tenants in roachtests. This will let us further stress the correctness of SQL Proxy as well as bridge the gap between how we test and how Serverless actually deploys.

The implementation of the SQL Proxy instance and directory should be fairly trivial, but there are a few decisions to be made around how mixedversion should interact with it. The core issue is that mixedversion tests are generally written to be deployment mode agnostic, such that we can metamorphically enable any one.

One immediate problem this poses is the fact that SQL Proxy does not allow connecting to a specific instance, but rather works as a load balancer, returning a "random" instance. While on paper this does seem to encourage less hardcoding of nodes, it introduces asymmetry between separate and non separate deployments, where we don't have this load balancer. It's also sometimes unavoidable to connect to certain instances e.g. we need to connect to each instance when doing our rolling restart.

One potential workaround for this is to have two ways to connect to instances:
1. If a random instance is requested, e.g. through `Query` or `RandomDB`, go through our proxy
2. If a specific instance is requested, we connect via the current existing connection pool, i.e. bypass the proxy

Another potentially simpler workaround might be to just spin up `len(instances)` connections, and query each one to see which node they connect to. We can then keep the current abstraction of connecting to a specific instance, while still having our connection go through SQL Proxy.

Whatever approach we end up on, it shouldn't ignore the fact that simply knowing which instance we are connected to is fairly important for debugging, i.e. even if we randomly pick instances we should log which node its connected to.

------

Another difference to consider stems from draining pods and the subsequent connection migration that happens. On one hand, this would let us actually test things concurrently the restarting of nodes. Currently, we only run test hooks in between the restarting of two nodes (spare the test or two that runs tpcc in the background with `--tolerate errors`). We could, if in separate process mode, allow the scheduling of test hooks concurrently with restart steps (and potentially FI steps).

However, there are a few catches to consider. Before restarting the node, we must first drain it on the director server and wait for connections to migrate. (From my understanding), this migration only happens on idle connections, so the migration time may be unbounded. There may be some test hooks that work with this requirement, but something like a workload probably won't ever be idle. This also greatly increases the time it takes for an upgrade to complete, which is not ideal considering some of the mixed version tests are already on the longer side.

Jira issue: CRDB-60772

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.