apache / apache/shardingsphere

Decoupling of data sources from sharding rule

Closed
#22,433 3 comments 0 reactions 0 assignees View on GitHub
feature: sharding type: discussion
Dominant language
Java
Stars
20.8k
Forks
6.9k
Avg merge
11h 35m
Merged PRs (30d)
326

Description

## Feature Request

### Is your feature request related to a problem?

When we add a new data source, the user has to update all the sharding rules to include the newly created data source.

**Example:**

```
CREATE SHARDING TABLE RULE t_orders (
DATANODES("ds_${0..1}.t_orders"),
DATABASE_STRATEGY(TYPE="standard", SHARDING_COLUMN=user_id, SHARDING_ALGORITHM(TYPE(NAME="inline",
PROPERTIES("algorithm-expression"="ds_${user_id%2}")))),
KEY_GENERATE_STRATEGY(COLUMN=record_id, KEY_GENERATOR=snowflake_key_generator)
);

CREATE SHARDING TABLE RULE t_invoices (
DATANODES("ds_${0..1}.t_invoices"),
DATABASE_STRATEGY(TYPE="standard", SHARDING_COLUMN=user_id, SHARDING_ALGORITHM(TYPE(NAME="inline",
PROPERTIES("algorithm-expression"="ds_${user_id%2}")))),
KEY_GENERATE_STRATEGY(COLUMN=record_id, KEY_GENERATOR=snowflake_key_generator)
);
```

When we add a new data source, `ds_2`, we need to update the same in both the `t_orders` and `t_invoices` sharding rules. It may look fine for a few tables, but it feels mundane for databases with hundreds of sharding rules.

### Describe the feature you would like.

The problem is mainly because of the tight coupling between the sharding rule and data sources. It's ideal to have some sort of indirection between the sharding rule and data sources.

**Proposal:**

Introduce a layer in the middle that can abstract the data sources.

![image](https://user-images.githubusercontent.com/13797616/204075844-b6136c08-31f1-43b2-8dea-d3dee680606c.png)

```
CREATE DATASOURCE GROUP t_orders_group (
DATANODES("ds_${0..1}")
);
```
```
CREATE SHARDING TABLE RULE t_orders (
DATASOURCE_GROUP("t_orders_group.t_orders"),
DATABASE_STRATEGY(TYPE="standard", SHARDING_COLUMN=user_id, SHARDING_ALGORITHM(TYPE(NAME="inline",
PROPERTIES("algorithm-expression"="ds_${user_id%2}")))),
KEY_GENERATE_STRATEGY(COLUMN=record_id, KEY_GENERATOR=snowflake_key_generator)
);
```

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points to begin with. First clarify the intended data-source-group syntax and implementation scope; done would mean sharding rules can reference a group whose data sources can be extended without editing each rule.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.