sequelize / sequelize/sequelize

Feature request: Implement sharding

Open
#3,806 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

hard status: in discussion
Dominant language
TypeScript
Stars
30.4k
Forks
4.3k
Avg merge
1d 6h
Merged PRs (30d)
68

Description

Hello. I want to request the sharding support to sequelize.

Currently I have a sharded databases which I need to support in my app, but this gives me a lot of trouble. I tried to use your framework but did not succeed. I will explain the architecture:
Suppose I have 3 databases, one is "meta db" which stores map of user data (i.e. on which shard the user data is) and two shards with equal structure storing user data.

var MetaUser = sequelize.define("MetaUser", {
        shardId: { type: DataTypes.INTEGER(10), allowNull: false }
    }
};

var User = sequelize.define("User", {
        name: { type: DataTypes.STRING, allowNull: false },
        about: { type: DataTypes.STRING(2048) },
    }
};

The problem is - all this databases can be on different servers so I need to create 3 different connections, or 3 different sequelize instances. And the following looks like (suppose dbConfig is read from json file and models are described in isolated files):

var metaConnection = new Sequelize(dbConfig.meta.database,
    dbConfig.meta.username,
    dbConfig.meta.password,
    dbConfig.meta.options);
module.exports.dbMeta = metaConnection;
var user1Connection = new Sequelize(dbConfig.shard1.database,
    dbConfig.shard1.username,
    dbConfig.shard1.password,
    dbConfig.shard1.options);
module.exports.dbUser1 = user1Connection;
var user2Connection = new Sequelize(dbConfig.shard2.database,
    dbConfig.shard2.username,
    dbConfig.shard2.password,
    dbConfig.shard2.options);
module.exports.dbUser2 = user2Connection;

module.exports.MetaUser = metaConnection.import(__dirname + "/models/MetaUser");
module.exports.User1 = user1Connection.import(__dirname + "/models/User");
module.exports.User2 = user2Connection.import(__dirname + "/models/User");

This gives me problems:

  1. I cannot reference one model from another in model's module since I don't have access to other model's connection in this model
  2. I want to use User model in my project, and have no need in MetaUser, but I have to handle User through MetaUser in order to calculate necessary shard connection.

If someone experienced similar problems, follow me to the solution, please.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how Sequelize instances and imported models are connected, using the MetaUser, User, and connection setup shown in the issue as the entry point. Define the required sharding behavior and verify it with tests covering shard lookup, cross-model references, and direct User access; the issue names no existing files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.