cockroachdb / cockroachdb/cockroach
ua: use `sql_instances` instead of gossip for dialing the address during the migration mixed mode
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
During our UA migration experiment, we observed that in a mixed-mode while running the TPCC benchmark on a node which is started with `TenantTwo` as system tenant, some data is written to wrong key-space. That is, a subset of warehouses were written to Prefixless keyspace when we expected all of that to be written to `/Tenant/2/*`.
Root cause of this is the import is happening on system tenant, while distributing the data for import on other nodes it uses the gossip information to find the node address to which it should connect to. In a mixed mode, this would be wrong since other node could still be `TenantOne` as system tenant which led to writing to wrong keyspace.
**Describe the solution you'd like**
Since `system.sql_instances` would have the right information even in mixed mode, the work done in https://github.com/cockroachdb/cockroach/pull/130235 should fix the issue. But before that we have to resolve the parity gap mentioned in https://github.com/cockroachdb/cockroach/pull/130235#issuecomment-2376034189
**Describe alternatives you've considered**
We can also consider improving the issue mentioned in https://github.com/cockroachdb/cockroach/issues/92524. This may have prevented this issue, as it would make use of single RPC listener for all tenants and client must pass the tenant id in connections.
**Additional context**
Mixed-mode: Cluster setup in which we are restarting nodes such that `TenantTwo` starts as system tenant. Not all nodes would come up at the same time and some nodes would still have `TenantOne` as system tenant.
Jira issue: CRDB-45183
Contributor guide
Research direction
Start by reading the mixed-mode migration behavior described here, the system.sql_instances lookup, and the work in PR 130235, including its noted parity gap. Compare that path with gossip-based address selection and issue 92524; done means imports consistently dial the correct tenant-aware address without writing to the wrong key-space in mixed mode.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100