change_stream update with truncatedArrays can be replayed as replacement and drop fields
- Dominant language
- Go
- Stars
- 1.8k
- Forks
- 466
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
When MongoShake reads incremental updates via `change_stream`, an update event can contain `updateDescription.truncatedArrays` with empty `updatedFields` and empty `removedFields`. This can happen for aggregation-pipeline updates that shrink an array.
The change stream conversion path currently appears to build the replay oplog object only from `updatedFields` and `removedFields`. If both are empty, the generated update object is empty. The executor then treats update objects without a `$` operator as replacement updates, so the target document can be replaced by an empty replacement document and end up preserving only `_id`.
This causes source/target divergence and target-side field loss.
### Environment
- MongoShake version observed: v2.8.4
- Code inspection suggests the same gap exists on current `develop` / recent v2.8.x code paths.
- Relevant mode/config shape:
- `sync_mode = incr`
- `incr_sync.mongo_fetch_method = change_stream`
- `incr_sync.change_stream.watch_full_document = false`
- `tunnel = direct`
### Minimal reproduction shape
Start with the same document on source and target:
```javascript
db.items.insertOne({
_id: 1,
arr: ["a", "b"],
keep: "value"
})
```
Apply an aggregation-pipeline update that truncates the array:
```javascript
db.items.updateOne(
{ _id: 1 },
[
{ $set: { arr: { $slice: ["$arr", 1] } } }
]
)
```
MongoDB may emit a change stream update event shaped like:
```javascript
{
operationType: "update",
documentKey: { _id: 1 },
updateDescription: {
updatedFields: {},
removedFields: [],
truncatedArrays: [
{ field: "arr", newSize: 1 }
]
}
}
```
### Expected behavior
MongoShake should preserve the other fields and apply the array truncation on the target:
```javascript
{ _id: 1, arr: ["a"], keep: "value" }
```
### Actual behavior
Because the change stream update conversion ignores `truncatedArrays`, the generated replay update object can be empty. The executor then follows the replacement path for update objects that do not contain a `$` operator, so the target document can become effectively:
```javascript
{ _id: 1 }
```
### Code path
The relevant current code paths appear to be:
- `oplog/change_stream_event.go`: update conversion handles `updatedFields` and `removedFields`, but not `updateDescription.truncatedArrays`.
- `executor/db_writer_single.go` and `executor/db_writer_bulk.go`: update objects without a `$` operator are replayed through replacement semantics.
- The oplog fetch path already has `$v:2` diff conversion logic for array truncation, but the change stream path bypasses that representation.
### Suggested fix
Possible fixes:
1. Parse `updateDescription.truncatedArrays` in the change stream update conversion path.
2. Convert each truncated array entry into a safe target update, for example a pipeline update using `$slice`, or an equivalent update expression that truncates the array without replacing the whole document.
3. Add a guard so an update event converted to an empty replay object cannot be silently executed as a replacement update.
4. Add a regression test for a change stream update whose only change is `updateDescription.truncatedArrays`.
MongoDB documents `updateDescription.truncatedArrays` as part of update change events, so MongoShake should either replay it correctly or fail safely instead of replacing the target document.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.