Automattic / Automattic/mongoose

Document Init Performance Bottleneck

Open
#12,108 6 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
JavaScript
Stars
27.5k
Forks
4k
Avg merge
2d 7h
Merged PRs (30d)
35

Description

### Prerequisites

- [X] I have written a descriptive issue title
- [X] I have searched existing issues to ensure the performance issue has not already been reported

### Last performant version

(last two/three years)

### Slowed down in version

5.10.18

### Node.js version

16.14.0

### 🦥 Performance issue

We query a large number of documents without using .lean() in some of our express.js GET endpoints.
Those documents contain some subdocument properties.

Our HTTP GET Endpoints spend about 83% of their time with synchronous operations on the CPU which block the event loop.
In particular the `Document.init` method has a large total time (see APM screenshot below).
The `clone` method has a large self time (see Tracing screenshot below).

**APM**:
![mongoose-doc-init-performance](https://user-images.githubusercontent.com/5209395/179064517-821735a6-d47a-4934-8059-46d3b2d2c6f5.png)

**Local Node Tracing:**
![mongoose-doc-init-performance-local-flame](https://user-images.githubusercontent.com/5209395/179064524-1a3b948d-a5cc-40ed-aaae-2263059b7c70.png)

![mongoose-doc-init-performance-4](https://user-images.githubusercontent.com/5209395/179099827-bf483b91-3160-46ac-bb16-e1e2b370e605.png)

### Steps to Reproduce

I am currently working on a small repo to reproduce this issue. Just wanted to post the timing analysis in advance in case someone has seen the same behaviour in APM tools.

### Expected Behavior

Document.init should not take that long. The performance is currently not acceptable. We have to use .lean() on all GET endpoints otherwise our endpoints are slowed down x30 (from 100ms to 3000ms on average).

Contributor guide

Open the contributing guide

Research direction

Start with the Document.init and clone methods named in the report, then review the APM and local tracing screenshots for the reported CPU cost. The reproduction repository is not yet provided, so first establish a reproducible case for large document queries with subdocuments. Done means reducing the synchronous initialization cost and avoiding the reported endpoint slowdown without requiring .lean().

Written by the indexing model from the issue text.

Assessment

Tech stack
express, javascript, mongodb, node.js
Domain
api, backend, databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.