Meteor-Community-Packages / Meteor-Community-Packages/denormalize

High memory usage and slow to update large numbers of parent documents

Open
#33 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
20
Forks
15
PR merge metrics
No merged PRs in 30d

Description

The underlying [meteor-collection-hooks](https://github.com/Meteor-Community-Packages/meteor-collection-hooks) performs two `fetch()`s on `update()` for any collection with an `after.update` hook defined.
- the [first](https://github.com/Meteor-Community-Packages/meteor-collection-hooks/blob/master/update.js#L38) to get the ids of all docs matching the selector
- the [second](https://github.com/Meteor-Community-Packages/meteor-collection-hooks/blob/9581d5bd63320e1e98a201adf68fd6f1fe6bfdcd/update.js#L80) to iterate over these docs post-update and fire the `after` hooks

This can be both slow for large numbers of docs, as well as expensive on memory as it is doing a `fetch()` of all docs rather than iterating over the cursor.

This `denormalize` package adds a [parentCollection.after.update hook](https://github.com/Meteor-Community-Packages/denormalize/blob/master/cache.js#L96), but also calls `parentCollection.update()` in any relevant `childCollection` mutation hook to maintain the caches. The result is that on _any_ child document mutation there is a chain of hooks causing the 2x `fetch()`, e.g.:
```
childCollection.update()
-> childCollection.after.update()
-> parentCollection.update()
-> parentCollection.after.update() // 2x fetch()
```

Related: https://github.com/Meteor-Community-Packages/meteor-collection-hooks/issues/259

### Suggestion

Denormalize simply needs to do an `parentCollection.updateMany()` to update the caches without the extra pre-fetching of ids the hooks do to support arbitrary selectors. Perhaps this package should wrap the Mongo API mutators directly, similar to how the hooks tie in, so as to avoid the chain of hook logic. The one downside I see is exactly that: _are there other hooks expected to be chained by the cache update, or even chained denormalization, which this would break?_ Perhaps this could be an opt-in alternative for maintaining the cache if the user does not require triggering hooks/chaining via the cache update.

Contributor guide

Open the contributing guide

Research direction

Start with cache.js at the parentCollection.after.update hook and the child collection mutation hooks that call parentCollection.update(). Then inspect the referenced update.js paths in meteor-collection-hooks to understand the two fetches. Done means reducing the memory and update cost for large parent sets while preserving any required hook or denormalization chaining.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, mongodb
Domain
backend, database, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.