Meteor-Community-Packages / Meteor-Community-Packages/denormalize
High memory usage and slow to update large numbers of parent documents
- Dominant language
- JavaScript
- Stars
- 20
- Forks
- 15
- PR merge metrics
- No merged PRs in 30d
Description
The underlying [meteor-collection-hooks](https://github.com/Meteor-Community-Packages/meteor-collection-hooks) performs two `fetch()`s on `update()` for any collection with an `after.update` hook defined.
- the [first](https://github.com/Meteor-Community-Packages/meteor-collection-hooks/blob/master/update.js#L38) to get the ids of all docs matching the selector
- the [second](https://github.com/Meteor-Community-Packages/meteor-collection-hooks/blob/9581d5bd63320e1e98a201adf68fd6f1fe6bfdcd/update.js#L80) to iterate over these docs post-update and fire the `after` hooks
This can be both slow for large numbers of docs, as well as expensive on memory as it is doing a `fetch()` of all docs rather than iterating over the cursor.
This `denormalize` package adds a [parentCollection.after.update hook](https://github.com/Meteor-Community-Packages/denormalize/blob/master/cache.js#L96), but also calls `parentCollection.update()` in any relevant `childCollection` mutation hook to maintain the caches. The result is that on _any_ child document mutation there is a chain of hooks causing the 2x `fetch()`, e.g.:
```
childCollection.update()
-> childCollection.after.update()
-> parentCollection.update()
-> parentCollection.after.update() // 2x fetch()
```
Related: https://github.com/Meteor-Community-Packages/meteor-collection-hooks/issues/259
### Suggestion
Denormalize simply needs to do an `parentCollection.updateMany()` to update the caches without the extra pre-fetching of ids the hooks do to support arbitrary selectors. Perhaps this package should wrap the Mongo API mutators directly, similar to how the hooks tie in, so as to avoid the chain of hook logic. The one downside I see is exactly that: _are there other hooks expected to be chained by the cache update, or even chained denormalization, which this would break?_ Perhaps this could be an opt-in alternative for maintaining the cache if the user does not require triggering hooks/chaining via the cache update.
Contributor guide
Research direction
Start with cache.js at the parentCollection.after.update hook and the child collection mutation hooks that call parentCollection.update(). Then inspect the referenced update.js paths in meteor-collection-hooks to understand the two fetches. Done means reducing the memory and update cost for large parent sets while preserving any required hook or denormalization chaining.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, mongodb
- Domain
- backend, database, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100