Automattic / Automattic/mongoose

Support for schema-driven deep population of all referenced paths without requiring explicit path specification

Open
#16,074 4 comments 0 reactions 0 assignees View on GitHub
enhancement new feature
Dominant language
JavaScript
Stars
27.5k
Forks
4k
Avg merge
2d 7h
Merged PRs (30d)
35

Description

### Prerequisites

- [x] I have written a descriptive issue title
- [x] I have searched existing issues to ensure the feature has not already been requested

### 🚀 Feature Proposal

Add native support for automatic, schema-driven deep population of all referenced paths without requiring explicit path specification. This would enable a "full populate" method (e.g., `Model.find().populateAll({ maxDepth: 3, exclude: ['sensitivePath'] })`) that inspects the schema for all `ref` fields, recursively populates them up to a configurable depth, and handles arrays/subdocs.

### Motivation

- Current support approaches require re-querying the document from the database after the transaction commit, which adds overhead and potential for race conditions if data changes meanwhile.
- Population paths are explicitly hardcoded ("posts", "comments", "author"), making the function brittle: Any schema changes (e.g., adding new refs like "tags" or deeper nesting) require manual updates to the code.
- No abstraction or automation: Lacks a generalized way to discover and populate all ref paths recursively based on the schema, leading to maintenance issues in complex, evolving models.
- Scalability concerns: For larger schemas with many levels of nesting or circular refs, maintaining these populate chains becomes error-prone and non-DRY (Don't Repeat Yourself).
- Inefficiency in transactions: Population isn't integrated with save/transaction ops, forcing separation of "save" and "populate" logic.

### Example

Reproducible example showing current problem space:

```javascript
const mongoose = require('mongoose');
const { Schema } = mongoose;

// 2. Define Models (minimal setup)
const Comment =
mongoose.models.Comment ||
mongoose.model(
'Comment',
new Schema({
text: String,
author: { type: Schema.Types.ObjectId, ref: 'User' },
})
);

const Post =
mongoose.models.Post ||
mongoose.model(
'Post',
new Schema({
title: String,
comments: [{ type: Schema.Types.ObjectId, ref: 'Comment' }],
})
);

const User =
mongoose.models.User ||
mongoose.model(
'User',
new Schema({
name: String,
posts: [{ type: Schema.Types.ObjectId, ref: 'Post' }],
})
);

async function buildFullPayload(userId) {
return await User.findById(userId).populate({
path: 'posts',
populate: {
path: 'comments',
populate: { path: 'author' },
},
});
}

async function main() {
try {
console.log('--- Establishing Connection ---');
await mongoose.connect('mongodb://localhost:27017/testdb'); // Adjust URI as needed

// 3. Sandbox Setup: Clear existing data to prevent ID collisions or bloat
console.log('--- Cleaning Sandbox ---');
await Promise.all([User.deleteMany({}), Post.deleteMany({}), Comment.deleteMany({})]);

const session = await mongoose.startSession();
session.startTransaction();

try {
console.log('--- Running Transaction ---');
const [user] = await User.create([{ name: 'Alice' }], { session });
const [post] = await Post.create([{ title: 'My Post' }], { session });
const [comment] = await Comment.create([{ text: 'Great post!', author: user._id }], { session });

post.comments.push(comment._id);
await post.save({ session });

user.posts.push(post._id);
await user.save({ session });

await session.commitTransaction();
console.log('Transaction Committed.');

// 4. Analysis: Log evidence of staleness
// First, show the in-memory user document post-commit (stale, with unpopulated refs as ObjectIDs)
console.log('--- Evidence of Staleness ---');
console.log('Stale User (in-memory post-commit):', JSON.stringify(user, null, 2));

// Basic test
const demoLogic = (doc, label) => {
console.log(`--- ${label} Check ---`);
// Try to access the first comment's author name
const post = doc.posts?.[0];
const firstComment = post?.comments?.[0];
const authorName = firstComment?.author?.name;

console.log(`Path: User -> Post -> Comment -> Author Name:`, authorName || '❌ NOT FOUND (Stale/Unpopulated)');
};

// 4.1. BEFORE population
demoLogic(user, 'Stale Object');

const fullPayload = await buildFullPayload(user._id);
console.log('Full Populated Payload (after buildFullPayload):', JSON.stringify(fullPayload, null, 2));

// 2. AFTER population
demoLogic(fullPayload, 'Populated Object');

// Log reasons why this version of buildFullPayload is unsatisfactory
console.log('--- Reasons Why buildFullPayload is Unsatisfactory ---');
console.log(
'- Requires re-querying the document from the database after the transaction commit, which adds overhead and potential for race conditions if data changes meanwhile.'
);
console.log(
'- Population paths are explicitly hardcoded ("posts", "comments", "author"), making the function brittle: Any schema changes (e.g., adding new refs like "tags" or deeper nesting) require manual updates to the code.'
);
console.log(
'- No abstraction or automation: Lacks a generalized way to discover and populate all ref paths recursively based on the schema, leading to maintenance issues in complex, evolving models.'
);
console.log(
"- Scalability concerns: For larger schemas with many levels of nesting or circular refs, maintaining these populate chains becomes error-prone and non-DRY (Don't Repeat Yourself)."
);
console.log('- Inefficiency in transactions: Population isn\'t integrated with save/transaction ops, forcing separation of "save" and "populate" logic.');
} catch (err) {
// Only abort if the transaction is still "In Progress"
if (session.transaction.isActive) {
await session.abortTransaction();
}
throw err;
} finally {
session.endSession();
}

// 5. Teardown: Drop collections to leave the DB as we found it
console.log('--- Tearing Down Sandbox ---');
await mongoose.connection.db.dropCollection('users');
await mongoose.connection.db.dropCollection('posts');
await mongoose.connection.db.dropCollection('comments');
console.log('Teardown complete.');
} catch (err) {
console.error('Test Failed:', err);
} finally {
await mongoose.disconnect();
console.log('Disconnected.');
}
}

main();
```

Run output:
```
node .\test1.js
PS C:\Users\myUser\workspaces\mongoose-issue-example> node .\test1.js
--- Establishing Connection ---
--- Cleaning Sandbox ---
--- Running Transaction ---
Transaction Committed.
--- Evidence of Staleness ---
Stale User (in-memory post-commit): {
"name": "Alice",
"posts": [
"69ab00955d4a38d6e1a210ed"
],
"_id": "69ab00955d4a38d6e1a210eb",
"__v": 1
}
--- Stale Object Check ---
Path: User -> Post -> Comment -> Author Name: ❌ NOT FOUND (Stale/Unpopulated)
Full Populated Payload (after buildFullPayload): {
"_id": "69ab00955d4a38d6e1a210eb",
"name": "Alice",
"posts": [
{
"_id": "69ab00955d4a38d6e1a210ed",
"title": "My Post",
"comments": [
{
"_id": "69ab00955d4a38d6e1a210ef",
"text": "Great post!",
"author": {
"_id": "69ab00955d4a38d6e1a210eb",
"name": "Alice",
"posts": [
"69ab00955d4a38d6e1a210ed"
],
"__v": 1
},
"__v": 0
}
],
"__v": 1
}
],
"__v": 1
}
--- Populated Object Check ---
Path: User -> Post -> Comment -> Author Name: Alice
--- Reasons Why buildFullPayload is Unsatisfactory ---
- Requires re-querying the document from the database after the transaction commit, which adds overhead and potential for race conditions if data changes meanwhile.
- Population paths are explicitly hardcoded ("posts", "comments", "author"), making the function brittle: Any schema changes (e.g., adding new refs like "tags" or deeper nesting) require manual updates to the code.
- No abstraction or automation: Lacks a generalized way to discover and populate all ref paths recursively based on the schema, leading to maintenance issues in complex, evolving models.
- Scalability concerns: For larger schemas with many levels of nesting or circular refs, maintaining these populate chains becomes error-prone and non-DRY (Don't Repeat Yourself).
- Inefficiency in transactions: Population isn't integrated with save/transaction ops, forcing separation of "save" and "populate" logic.
--- Tearing Down Sandbox ---
Teardown complete.
Disconnected.
```

Proposed API (first thoughts):

- `Query.prototype.populateAll(options)` or similar, where `options` could include:
-- `maxDepth`: Limit recursion (default: 5) to prevent infinite loops (e.g., circular refs).
-- `exclude`: Array of paths to skip.

Is not a fully solved problem due to the `maxDepth` limitation, but it would be an improvement.

### Related Issues
#8153
-- In my opinion, the above issue was Closed without sufficient debate and/or explanation

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the proposed Query.prototype.populateAll entry point and the existing buildFullPayload example, then read related issue #8153. Define how schema-discovered refs, maxDepth, exclude, arrays, subdocuments, circular references, and transaction behavior should work; the feature is done when those rules are agreed and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, mongodb, node.js
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.