airbytehq / airbytehq/airbyte

[source-mongodb-v2] CDC resume token timestamp frozen — incremental sync replays 101 records/run forever (initial_waiting_seconds workaround from #71046 ineffective)

オープン
#80,250 コメント 10 件 リアクション 0 件 担当者 0 名 GitHub で見る
autoteam community connectors/source/mongodb-v2 team/use
主要言語
Python
スター
22.1k
フォーク
5.4k
平均マージ
5時間
マージ済み PR(30日)
671

説明

### Connector Name

source-mongodb-v2

### Connector Version

2.0.7 (also reproduced on 2.0.4 — see #71046)

### What step the error happened?

During the sync

### Relevant information

This is a follow-up to #71046. That thread's recommended workaround — raise `initial_wait_time` / `initial_waiting_seconds` so Debezium doesn't time out on subsequent batches — **does not resolve the problem for us.** We run with `initial_waiting_seconds: 1200` (and `discover_timeout_seconds: 1200`), yet incremental CDC sync **loads exactly 101 records every run, indefinitely**, until a full refresh — after which the cycle returns within hours. We're now on connector **2.0.7**, newer than the 2.0.4 in the original report, and the behavior is unchanged.

**Root cause appears to be a stuck CDC resume token, not a timeout.** Diagnosing across four consecutive scheduled syncs, the saved Debezium offset's **timestamp component is frozen at a single oplog second** (`sec=1781748186`, the instant the last full load completed). Only the *ordinal* within that one second advances (~101 events/run), while the live oplog moves on by hours:

| Job | Resumed offset | Live oplog token ts | Rows synced |
|-----|----------------|---------------------|-------------|
| 4255 | `{sec=1781748186, ord=2864}` | 1781757158 | 101 |
| 4259 | `{sec=1781748186, ord=3066}` | 1781758834 | 101 |
| 4266 | `{sec=1781748186, ord=3471}` | 1781762500 | 101 |
| 4272 | `{sec=1781748186, ord=3572}` | 1781763326 | 101 |

The connector is permanently pinned to one oplog instant, draining ~101 oplog entries per run and never advancing the resume-token timestamp, so it can never catch up to real time. The `101` is MongoDB's default change-stream cursor first-batch size: each run reads one batch, advances the ordinal slightly, waits out the no-records timeout, commits a resume token whose `sec` never moves, and exits. A full refresh resets the token to "now," which works until the same freeze recurs.

Note also that despite `initial_waiting_seconds: 1200` ("Subsequent cdc record wait time: PT20M"), the record iterator closes on a separate **600s** no-records timeout — so the `initial_wait_time` workaround doesn't govern this path at all.

**Environment**
- Airbyte: v2.0.1 (self-hosted community, abctl)
- source-mongodb-v2: 2.0.7
- Embedded Debezium: 2.6.2.Final
- Destination: Snowflake
- MongoDB: Atlas 8.0.26, `ATLAS_REPLICA_SET` (4-node), read from secondary (`readPreference=secondary&readPreferenceTags=nodeType:READ_ONLY`)
- Source config: `update_capture_mode: Lookup`, `initial_waiting_seconds: 1200`, `discover_timeout_seconds: 1200`, `initial_load_timeout_hours: 24`, `invalid_cdc_cursor_position_behavior: Fail sync`
- Stream: single collection, `incremental_deduped_history`, cursor `_ab_cdc_cursor`, every 45 min

### Relevant log output

```shell
source ... MongoDbCdcInitializer(createCdcIterators):80 Subsequent cdc record wait time: PT20M seconds
source ... MongoDbCdcInitializer(createCdcIterators):277 Initial load has finished completely - only reading the oplog
source ... AirbyteDebeziumHandler getIncrementalIterators: Using CDC: true / DBZ version: 2.6.2.Final
source ... MongoDbConnection(validateLogPosition):192 Found existing offset for at {sec=1781748186, ord=3572, resume_token=826A3351DA00000DF4...}
source ... ChangeEventSourceCoordinator(streamEvents):279 Starting streaming
source ... DebeziumRecordIterator(requestClose):275 No records were returned by Debezium in the timeout seconds 600, closing the engine and iterator
source ... SourceStateIterator(computeNext):84 sending final state message, with count per stream: {miter-production_ledgerlineitems=101}
source ... IntegrationRunner stopOrphanedThreads:469 The main thread is exiting while children non-daemon threads from a connector are still active.
Active non-daemon thread info: debezium-mongodbconnector-...-replicator-fetcher-0
replication-orchestrator Sync summary: { "status":"completed", "recordsSynced":101, ... }
```

A likely-related secondary warning on every run (oplog stats casting bug):

```shell
source ... MongoDbCdcInitializer(logOplogInfo):356 Unable to query for op log stats, exception: class java.lang.Integer cannot be cast to class java.lang.Long
```

### Asks

1. Why does the committed resume token's timestamp stay pinned to a single oplog second while only the ordinal advances?
2. Is reading from a secondary (`nodeType:READ_ONLY`) with `update_capture_mode: Lookup` implicated?
3. A real fix rather than "increase `initial_waiting_seconds`" — that workaround is already applied and does not help.

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12917

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start at MongoDbCdcInitializer.createCdcIterators, MongoDbConnection.validateLogPosition, and DebeziumRecordIterator.requestClose; inspect how the resume offset is validated, committed, and closed after the 600-second no-records timeout. Reproduce consecutive CDC runs with the supplied offsets and logs, including secondary reads and Lookup mode. Done means the committed resume-token timestamp advances with the oplog and repeated runs no longer replay 101 records.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java, mongodb
領域
data-engineering, databases
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
静か
明瞭さ
説明が足りない
初心者へのやさしさ
38/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。