Introduce Meta Service Group to TiDB
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
# Support Meta Service Abstraction and Meta Service Group Routing
## Background
In TiDB distributed deployments, PD's etcd acts as the core metadata store and carries critical cluster workflows. In a multi-tenant architecture, etcd data for different TiDB clusters is isolated mainly by etcd key prefixes, which means each tenant keyspace effectively owns its own etcd key set.
In addition to the etcd keys accessed directly by TiDB, some etcd keys used inside PD also grow with the number of keyspaces. As business scale increases exponentially and the number of keyspaces continues to expand, the current architecture faces several challenges:
1. Storage and performance bottlenecks: a single etcd cluster is increasingly unable to absorb metadata storage demands and frequent read/write pressure at large keyspace scale, resulting in higher latency and saturated resource usage.
2. Limited scalability: a centralized metadata storage model does not satisfy the elastic scaling requirements of multi-tenant cloud deployments, which constrains cluster scale and business growth.
3. Stability risks: concentrated metadata storage increases the blast radius of failures, and abnormal operations from one keyspace may cause global service impact.
The current architecture is no longer sufficient for high-concurrency, low-latency metadata management in cloud multi-tenant scenarios. A distributed storage-layer redesign plus access-layer decoupling is needed to reduce performance pressure and improve both stability and scalability.
## Concepts
1. Meta Service
1. The concept of `Meta Service` replaces the old etcd-centric view. Existing etcd usage is essentially providing cluster-level metadata service, so it should be abstracted as `Meta Service`.
2. Implementation of Meta Service
1. `etcd`
2. Support other implementations in the future
3. Meta Service Group
1. If Meta Service is split into multiple clusters and each Meta Service cluster serves multiple keyspaces, then all keyspaces served by the same Meta Service cluster belong to the same `Meta Service Group`.
2. Mapping relationship: one etcd cluster = one Meta Service Group = multiple keyspaces.
4. PD Addrs
1. `PD Addrs` refers to PD API addresses.
## Goals
### 1. System Stability
1. Avoid mutual interference between the PD API server and the TiDB Meta Service path.
2. Isolate Meta Service operations across different keyspaces so that keyspaces do not affect each other. A failure in one Meta Service cluster should not impact other Meta Service clusters.
### 2. Cost Optimization
1. Meta Service scaling should no longer depend on PD resource evaluation or PD API request pressure. Only metadata storage cost and workload need to be considered, which avoids over-provisioning Meta Service resources for unrelated load.
### 3. Code Decoupling
1. Replace the existing TiDB direct-to-etcd access path with a Meta Service interface model, using `etcd` as the default implementation first.
2. The interface model should allow the Meta Service implementation to be replaced by other KV stores in the future, which may reduce storage cost.
3. The interface layer should make it easier to add observability and optimize resource usage.
### 4. Performance Optimization
1. The Meta Service Group architecture reduces read/write pressure on a single Meta Service cluster by distributing the original single-etcd load across multiple Meta Service clusters.
2. More workload-appropriate KV backends can be selected to optimize specific operations and make performance bottlenecks easier to identify.
### 5. Elastic Scalability
1. Each Meta Service cluster can scale independently to support dynamic resource scheduling for different business lines. For example, Kubernetes StatefulSet based scaling can support elastic expansion without service interruption.
2. Different business types can use different Meta Service clusters.
3. Different users can use different Meta Service clusters.
4. Meta Service cluster nodes can be increased or decreased based on group pressure.
5. Multiple KV backends can be supported to improve architectural flexibility and adapt to different workload requirements.
### 6. Operational Efficiency
1. The impact scope of a single Meta Service cluster failure becomes smaller, making issues easier to locate and repair.
2. Operational actions on a Meta Service cluster can focus only on that cluster itself and do not need to consider the PD API server path.
3. After introducing the interface layer on the TiDB side, metrics can be added in the implementation to improve diagnosis efficiency.
## Current Architecture
The current architecture uses PD's embedded etcd both for PD-internal metadata and for metadata accessed by TiDB keyspaces. As the number of keyspaces grows, more TiDB-side etcd access accumulates on the same metadata plane.
The original PDF presents this section as an architecture diagram showing:
1. PD services such as TSO service and scheduler service connected to the PD etcd cluster.
2. Multiple TiDB keyspaces directly using etcd clients.
3. Metadata access from many keyspaces converging on the same etcd cluster.
## Architecture Improvement
### Etcd Group Implementation
The proposed improvement introduces grouped metadata clusters. Instead of having all keyspaces access one shared metadata cluster, different groups of keyspaces are mapped to different Meta Service clusters.
The original PDF presents this section as a diagram illustrating:
1. A PD cluster that still serves PD-specific functions.
2. Multiple independent metadata clusters, each representing one Meta Service Group.
3. TiDB keyspaces accessing metadata through a Meta Service interface rather than directly binding to one shared etcd endpoint.
## Architecture Changes
### 1. Before
1. PD's etcd contains both global keys and keyspace-specific TiDB keys.
2. TiDB-related etcd usage inside PD increases as the number of keyspaces grows.
### 2. After
1. One TiKV cluster can have multiple Meta Service Groups.
2. Each Meta Service Group can serve multiple keyspaces.
3. The TiDB-side etcd client is replaced with a Meta Service client, which accesses the target Meta Service according to the Meta Service Group address.
## Code Changes
### Meta Service Code Design
The TiDB codebase should introduce a Meta Service abstraction layer for metadata operations currently tied to direct etcd access. The first implementation can continue to use `etcd`, but the interface should make backend replacement and grouped routing possible.
Suggested implementation direction:
1. Define a Meta Service client interface for metadata operations currently performed through direct etcd access.
2. Provide an `etcd` implementation as the initial backend for backward compatibility.
3. Add keyspace-to-group resolution and group-to-address resolution.
4. Route metadata requests through the Meta Service client based on the resolved Meta Service Group.
5. Keep single-group behavior compatible with existing deployments where applicable.
## Observability
Monitoring should be added in the Meta Service client interface to track Meta Service request size, operations, and related metadata dimensions.
Contributor guide
Assessment
This issue has not been assessed yet.