SharedChangeStream should support oversized change events with $changeStreamSplitLargeEvent
- Dominant language
- JavaScript
- Stars
- 44.8k
- Forks
- 5.2k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 25
Description
I'm submitting this PR as a follow up to #14763, and I suggest using this approach as the primary implementation with the suggestions in #14763 as the safety net.
### Version of Meteor showing the problem
Meteor 3.5.1 (`packages/mongo/shared_change_stream.js`).
### Last version where the problem did not occur
Unknown. The issue affects the change-stream-based Mongo driver in Meteor 3.5.1.
### Operating system
Any server-side operating system; the failure is caused by MongoDB's BSON size limit.
### Expected behavior
`SharedChangeStream` should support MongoDB change events whose materialized event exceeds 16 MB, where the MongoDB deployment supports `$changeStreamSplitLargeEvent` (MongoDB 7.0+). Large events should be fragmented by MongoDB, reassembled by the client before being dispatched to drivers, and resumed correctly.
For MongoDB versions without this stage, an oversized-event failure should not cause an endless restart loop. The stream should drop the unusable resume token, reopen from a fresh operation time, and reconcile attached drivers.
### Actual behavior
A change stream fails with an oversized BSON event:
```text
MongoServerError: PlanExecutor error during aggregation :: caused by ::
BSONObj size: 17701268 (0x10E1994) is invalid. Size must be between 0 and 16793600(16MB)
First element: _id: { _data: "826AA99D52000000982B042C0100296E5A1004…" }
```
`SharedChangeStream` requests both full images:
```js
const changeStreamOptions = {
fullDocument: 'updateLookup',
fullDocumentBeforeChange: 'whenAvailable',
};
```
A document below MongoDB's 16 MB document limit can still produce an oversized change event because the event includes the resume token, operation metadata, the post-image, and potentially the pre-image. A small update to a large document can therefore exceed the 16 MB event limit.
The error is raised while MongoDB materializes the change-stream event, before a normal user pipeline stage can reduce its size. Consequently, adding a `$project` stage alone does not solve the problem.
The current `_isNonResumableError()` does not classify `BSONObjectTooLarge` (code 10334) as non-resumable. The stored resume token is therefore retained, and the restart reopens with `startAfter` at the same oversized event. The stream repeatedly fails on the same event, observers become stale, and the application logs repeated change-stream restart errors.
### Steps to reproduce
1. Run a Meteor 3.5.1 application against a MongoDB replica set.
2. Create a collection and attach an observer so `SharedChangeStream` is opened.
3. Insert a document large enough that its change event, including full images and metadata, exceeds 16 MB but the document itself is still valid.
4. Update a small field on that document.
5. Observe the `BSONObj size ... is invalid` error and repeated restart attempts.
### Proposed solution
1. For MongoDB 7.0 and later, append `$changeStreamSplitLargeEvent` as the final stage of the change-stream watch pipeline. This stage must be applied to the server-side change-stream pipeline so MongoDB fragments oversized events before the 16 MB BSON limit is exceeded.
2. Reassemble fragments in `SharedChangeStream` before calling `_onChange()` on any driver. A fragment must not be independently matched, projected, or used to advance driver state.
3. Track resume tokens carefully: only the token for the completed logical event should become the stream's restart token. Restart behavior during an incomplete fragmented event must not cause duplicate or partial dispatch.
4. Gate use of `$changeStreamSplitLargeEvent` by MongoDB server capability/version, because older MongoDB versions reject the stage.
5. Retain a fallback for older versions and events that cannot be split. In particular, classify `BSONObjectTooLarge` / error code 10334 as non-resumable, drop the resume token, reopen from a fresh operation time, and invoke the existing driver resynchronization path.
6. Consider avoiding `fullDocumentBeforeChange: 'whenAvailable'` when pre-images are not required, since requesting both full images increases event size substantially.
The existing empty-pipeline invariant should also be documented or adjusted: `$changeStreamSplitLargeEvent` does not filter events, but it changes one logical change into multiple wire-level fragments. Fragment handling must occur before the existing per-driver filtering and operation-time advancement logic.
Contributor guide
Research direction
Start in packages/mongo/shared_change_stream.js, tracing changeStreamOptions, _isNonResumableError(), _onChange(), resume-token handling, and the existing per-driver filtering and resync path. Reproduce the BSON size failure against MongoDB 7.0+ and an older server, then verify that split fragments are reassembled before dispatch, only completed-event tokens advance, and the fallback avoids restart loops.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, mongodb
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100