matrixorigin / matrixorigin/matrixone

[Feature Request]: Sharding for partition table

Open
#16,438 1 comment 0 reactions 1 assignee Claimed by @iamlinjunhong View on GitHub
area/partition kind/feature kind/tech-request priority/p0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same feature request?

- [X] I have checked the existing issues.

### Is your feature request related to a problem?

```Markdown
MO's current load balancing is handled on the Proxy, which does not parse SQL, and the current load balancing strategy is to ensure that the number of sessions on each CN is balanced as much as possible.

The problem is that all CNs may be able to handle all the Partitions of T1 table, which brings the following serious problems:

1. Each CN subscribes to all the Partitions of table T1.
2. Each CN consumes the LogTail of all the Partitions it is subscribed to.
3. Each CN's Mem, Disk's Cache will store the data of all Partitions.
```

### Describe the feature you'd like

As the load increases and the MO cluster slowly gets bigger, for writes, the expected result should be something like this:

1. Each CN subscribes to a different Partition (read scalability can be scaled by multiple CNs subscribing to the same Partition to scale the read performance of a single Partition)
2. The operation to read the Committed data of a Partition should be invoked at the CN that subscribes to that Partition.
3. Partitions should be balanced across CNs in the cluster, e.g., if a CN expands or locks up, the subscribed partitions can be changed across CNs.

### Describe implementation you've considered

_No response_

### Documentation, Adoption, Use Case, Migration Strategy

_No response_

### Additional information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.