sequelize / sequelize/sequelize
Feature request: Implement sharding
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 30.4k
- Forks
- 4.3k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 68
Description
Hello. I want to request the sharding support to sequelize.
Currently I have a sharded databases which I need to support in my app, but this gives me a lot of trouble. I tried to use your framework but did not succeed. I will explain the architecture:
Suppose I have 3 databases, one is "meta db" which stores map of user data (i.e. on which shard the user data is) and two shards with equal structure storing user data.
var MetaUser = sequelize.define("MetaUser", {
shardId: { type: DataTypes.INTEGER(10), allowNull: false }
}
};
var User = sequelize.define("User", {
name: { type: DataTypes.STRING, allowNull: false },
about: { type: DataTypes.STRING(2048) },
}
};
The problem is - all this databases can be on different servers so I need to create 3 different connections, or 3 different sequelize instances. And the following looks like (suppose dbConfig is read from json file and models are described in isolated files):
var metaConnection = new Sequelize(dbConfig.meta.database,
dbConfig.meta.username,
dbConfig.meta.password,
dbConfig.meta.options);
module.exports.dbMeta = metaConnection;
var user1Connection = new Sequelize(dbConfig.shard1.database,
dbConfig.shard1.username,
dbConfig.shard1.password,
dbConfig.shard1.options);
module.exports.dbUser1 = user1Connection;
var user2Connection = new Sequelize(dbConfig.shard2.database,
dbConfig.shard2.username,
dbConfig.shard2.password,
dbConfig.shard2.options);
module.exports.dbUser2 = user2Connection;
module.exports.MetaUser = metaConnection.import(__dirname + "/models/MetaUser");
module.exports.User1 = user1Connection.import(__dirname + "/models/User");
module.exports.User2 = user2Connection.import(__dirname + "/models/User");
This gives me problems:
- I cannot reference one model from another in model's module since I don't have access to other model's connection in this model
- I want to use User model in my project, and have no need in MetaUser, but I have to handle User through MetaUser in order to calculate necessary shard connection.
If someone experienced similar problems, follow me to the solution, please.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how Sequelize instances and imported models are connected, using the MetaUser, User, and connection setup shown in the issue as the entry point. Define the required sharding behavior and verify it with tests covering shard lookup, cross-model references, and direct User access; the issue names no existing files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100