cockroachdb / cockroachdb/cockroach

ua: use `sql_instances` instead of gossip for dialing the address during the migration mixed mode

Open
#136,666 0 comments 0 reactions 0 assignees View on GitHub
A-multitenancy C-enhancement T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

During our UA migration experiment, we observed that in a mixed-mode while running the TPCC benchmark on a node which is started with `TenantTwo` as system tenant, some data is written to wrong key-space. That is, a subset of warehouses were written to Prefixless keyspace when we expected all of that to be written to `/Tenant/2/*`.

Root cause of this is the import is happening on system tenant, while distributing the data for import on other nodes it uses the gossip information to find the node address to which it should connect to. In a mixed mode, this would be wrong since other node could still be `TenantOne` as system tenant which led to writing to wrong keyspace.

**Describe the solution you'd like**

Since `system.sql_instances` would have the right information even in mixed mode, the work done in https://github.com/cockroachdb/cockroach/pull/130235 should fix the issue. But before that we have to resolve the parity gap mentioned in https://github.com/cockroachdb/cockroach/pull/130235#issuecomment-2376034189

**Describe alternatives you've considered**

We can also consider improving the issue mentioned in https://github.com/cockroachdb/cockroach/issues/92524. This may have prevented this issue, as it would make use of single RPC listener for all tenants and client must pass the tenant id in connections.

**Additional context**

Mixed-mode: Cluster setup in which we are restarting nodes such that `TenantTwo` starts as system tenant. Not all nodes would come up at the same time and some nodes would still have `TenantOne` as system tenant.

Jira issue: CRDB-45183

Contributor guide

Open the contributing guide

Research direction

Start by reading the mixed-mode migration behavior described here, the system.sql_instances lookup, and the work in PR 130235, including its noted parity gap. Compare that path with gossip-based address selection and issue 92524; done means imports consistently dial the correct tenant-aware address without writing to the wrong key-space in mixed mode.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.