debezium / debezium/dbz

MongoDB startup validation rejects valid split-event resume tokens

Open
#2,619 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Bug report

The MongoDB connector can reject a valid resume token during startup after processing a split Change Stream event, even when the connector configuration and source document remain unchanged.

**What Debezium connector do you use and what version?**

MongoDB source connector. Reproduced with 3.6.2.Final and main at e1ab11652cc566cfc036c5cb1d8e196a0d8468bf (3.7.0-SNAPSHOT).

**What is the connector configuration?**

Relevant settings:

```properties
snapshot.mode=no_data
capture.mode=change_streams_update_full
cursor.oversize.handling.mode=split
heartbeat.interval.ms=0
```

Startup token validation remains enabled. Heartbeats are disabled to keep the committed offset at the completed split event's final fragment.

**What is the captured database version and mode of deployment?**

MongoDB 8.0.21, single-node replica set in Docker. Regression tests use an actual MongoDB container and Debezium AsyncEmbeddedEngine.

**What behavior do you expect?**

Startup validation should use the same full-document and pre-image options as streaming and accept a token that MongoDB can resume from with those options. The connector should restart and capture subsequent changes.

**What behavior do you see?**

Streaming applies fullDocument=updateLookup, but the startup validation cursor omits that option. A split update event can therefore become too small to split when MongoDB reconstructs it for validation. Resuming from its fragment token fails with MongoDB error 280:

```text
Attempted to resume from a split event fragment, but the event in the resumed stream was not large enough to be split
```

Debezium treats the token as invalid and fails to start with snapshot.mode=no_data.

The same mismatch affects capture modes requesting stored post-images or pre-images: fullDocument and fullDocumentBeforeChange are configured in the streaming path, but omitted from startup validation.

**Do you see the same behaviour using the latest released Debezium version?**

Yes, reproduced on 3.6.2.Final and the main commit above. The parameterized regression tests and local fix were validated on main.

**Do you have the connector logs, ideally from start till finish?**

The initial reproduction reports the saved offset as unavailable during startup. A local regression test verifies that an independent MongoDB cursor can resume from the persisted token with matching document options, then asserts that Debezium's validateLogPosition() accepts that same token. Before the fix, all five parameterized cases fail at:

```text
[Startup validation must accept the committed token of a completed split event]
Expecting value to be true but was false
```

After sharing the document-option configuration between cursor creation paths, all five cases pass, including connector restart and consumption of a subsequent insert.

**How to reproduce the issue using our tutorial deployment?**

A minimal replica-set reproduction is sufficient; the regression test uses Engine/Testcontainers rather than the full Kafka tutorial deployment:

1. Before starting the connector, insert a document with a 9 MiB payload:

```javascript
db.getSiblingDB("dbit").splitEvents.insertOne({
_id: 1,
payload: "a".repeat(9 * 1024 * 1024)
});
```

2. Start the connector with the settings above, capturing dbit.splitEvents, and wait for streaming to begin.
3. Update the payload:

```javascript
db.getSiblingDB("dbit").splitEvents.updateOne(
{ _id: 1 },
{ $set: { payload: "b".repeat(9 * 1024 * 1024) } }
);
```

4. The update details and fullDocument are each approximately 9 MiB, producing two fragments. Wait for Debezium to emit the merged event and commit the last fragment's token.
5. Stop and restart the connector without changing the source document or configuration.
6. Startup token validation rejects the saved token. Opening a native cursor with the same token, split stage, and fullDocument=updateLookup succeeds.

When reproducing through Kafka Connect, broker, topic, and producer record-size limits must also permit the merged record. The Engine regression test avoids this separate transport limit.

**Investigation and proposed fix**

[MongoDbConnection.isValidResumeToken()](https://github.com/debezium/debezium/blob/e1ab11652cc566cfc036c5cb1d8e196a0d8468bf/debezium-connector-mongodb/src/main/java/io/debezium/connector/mongodb/connection/MongoDbConnection.java#L208) creates a Change Stream through MongoUtils.openChangeStream() and sets resumeAfter. Document options are applied separately in MongoDbStreamingChangeEventSource.initChangeStream().

Apply those document options in the shared Change Stream creation path so validation and streaming use the same event representation.

The local regression test covers full-document lookup, stored post-image, pre-image alone, and both full-document modes combined with pre-image. It checks actual two- or three-fragment events, the merged record, and the persisted final-fragment token before validating and restarting.

Related to debezium/dbz#64, which prompted this investigation. This issue specifically covers the startup validation mismatch. It does not claim to resolve the original streaming failure in #64 or failures caused by a change in the document returned by updateLookup between runs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.