Automattic / Automattic/monk

Stream aggregation results

Open
#237 3 comments 4 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
1.8k
Forks
180
PR merge metrics
No merged PRs in 30d

Description

Would it be possible to add an .each() method to aggregation results similar to that found on `find`. If we could stream each result that would help a lot with out of memory problems on big result sets.

Looking at the native mongodb driver, it seems as though data can be streamed from the cursor created by `var cursor = collection.aggregate()` via `cursor.on('data')`. I'm not really familiar with the internals of eaither driver, but maybe that could be exposed via `.each()`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how aggregation results are created and how the existing find().each() API exposes MongoDB cursors. Compare that path with the native aggregate cursor and its cursor.on('data') events. Done means aggregation results can be consumed one at a time without loading the full result set into memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, mongodb
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.