Automattic / Automattic/mongoose

Massive memory usage when creating a document with approximately 30k+ subdocuments

Open
#11,541 8 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
JavaScript
Stars
27.5k
Forks
4k
Avg merge
2d 7h
Merged PRs (30d)
35

Description

**Do you want to request a *feature* or report a *bug*?** Bug

**What is the current behavior?**

When saving a large document (9.66MB) I am getting what looks to be some sort of memory leak. I initially thought that this might just be a result of my document being too big but I have concluded that the behaviour I am seeing does seem to constitute a bug as far as I can tell at the moment. Perhaps it would be a good idea not to have such a large document but I feel that what I am trying to do should work, at least at this scale.

This document contains a large array of subdocuments (low hundreds). Updating the document takes a long time (about 50 seconds for this particular 9.66MB I am using as an example). During this time, memory usage as observed in the Chrome profiler jumps from around 65MB to around 500MB. The document does appear to be saved successfully, it just takes a long time and uses a lot of memory.

I am finding it a little difficult to debug this error so do let me know what information I can give to be more helpful.

I have watched the memory change over time and recorded the flow through my function and I am pretty sure that it is specifically the .save() function on a document instance which is taking ~50 seconds and that the memory increase is happening after the call to save. I do not have any hooks defined on the model.

**If the current behavior is a bug, please provide the steps to reproduce.**

**What is the expected behavior?**

Much less memory usage and quicker save.

**What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version.**
Mongoose: 6.2.7, Node v14.17.5, MongoDB: 5.0.6

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. First reproduce saving the approximately 9.66 MB document with about 30k subdocuments using Mongoose 6.2.7, Node 14.17.5, and MongoDB 5.0.6, then profile the document save; done means the save completes substantially faster with much lower memory growth.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, mongodb, node.js
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.