airbytehq / airbytehq/airbyte

[source-mongodb-v2] Saved Resume Token Time Regression Resulting in Missing Data

Open
#69,102 0 comments 0 reactions 0 assignees View on GitHub
area/connectors autoteam community connectors/source/mongodb-v2 needs-triage team/extensibility team/use type/bug
Dominant language
Python
Stars
22.1k
Forks
5.3k
PR merge metrics
PR metrics pending

Description

### Connector Name

source-mongodb-v2

### Connector Version

2.0.4

### What step the error happened?

Saving resume token

### Relevant information

In our Airbyte the MongoDB source connector is experiencing a critical issue where saved resume tokens at the end of the sync have timestamps earlier than initial resume tokens, resulting in missing data during CDC synchronization. Any cause as to why?

## Environment
- **Airbyte Version**: 1.7.1 self hosted community
- **Connector**: source-mongodb-v2
- **MongoDB Version**: 2.0.4

## Problem Description

### Issue 1: Complete Data Loss During Sync Window
**Time Period**: 2 weeks Oct 16 - Present

**Evidence**:
During a 9 hour timeframe we saw
- **MongoDB Oplog**: 1,623+ operations recorded for affected collection
- **Target Database**: 0 records synced for the same period
- **Specific Missing Records**: Documents inserted during the sync window are completely absent from target
- In another connection we had the mongodb source connector repeatedly ingesting 101 records each sync for 2 weeks straight.

### Issue 2: Resume Token Temporal Regression
**Observed Behavior**:
- Initial resume token timestamp: `Recent date/time`
- Saved resume token timestamp: `Date ~5 days earlier`
- **Result**: Saved token regresses to significantly earlier timestamp

### Issue 3: Consistent Batch Size Pattern
**Pattern**: Since mid-October, certain collection syncs consistently return exactly 101 documents
- This matches MongoDB's default first batch size
- When we looked at the logs we saw that the resume token was either:
progressing by mere seconds
outputted resume tokens were from those of oct 23! This suddenly self healed without any intervention and now it is stuck on Oct 28. It is Oct 30th on the day of writing this.

## Expected Behavior
1. Resume tokens should progress chronologically forward
2. All oplog operations should be captured and synced
3. Batch operations should complete with `getMore` calls when needed

## Actual Behavior
1. Resume tokens regress to earlier timestamps
2. Data operations are missed entirely during certain time windows
3. Syncs appear to "self-heal" but then revert to problematic behavior

## Impact
- **Data Loss**: Critical business data not reaching the data warehouse
- **Data Integrity**: Inconsistent state between source and target

---

**Priority**: Critical - Data Loss Issue
**Labels**: bug, mongodb, cdc, data-loss, resume-token, production

### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.