debezium / debezium/dbz

Provide a lossless handling mode for MongoDB BSON Timestamp values

Open
#2,570 0 comments 0 reactions 0 assignees View on GitHub
component/mongodb-connector type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Problem

`ExtractNewDocumentState` converts a MongoDB BSON Timestamp into a Kafka Connect Timestamp using only the BSON `time` component.

The BSON `increment` component is silently discarded.

As a result, distinct BSON values such as `Timestamp(1710000000, 7)` and `Timestamp(1710000000, 8)` produce the same transformed value.

This prevents consumers from reconstructing the original BSON value and can collapse values that are distinct in MongoDB.

## Environment

* Percona Server for MongoDB 8.0.29-13
* Debezium MongoDB Source 3.7.0-SNAPSHOT
* Kafka 3.8.0

The same behavior was reproduced with Debezium 3.6.1.Final.

## Reproduction

Configure the connector with `ExtractNewDocumentState` and insert:

```javascript
db.timestamp_values.insertMany([
{
_id: "timestamp-1",
value: Timestamp(1710000000, 7)
},
{
_id: "timestamp-2",
value: Timestamp(1710000000, 8)
}
]);
```

## Actual behavior

Both BSON values are converted to the same Connect Timestamp representing:

```text
2024-03-09T16:00:00Z
```

The increment values `7` and `8` are not present in the schema or payload.

The conversion currently behaves equivalently to:

```java
new Date(1000L * value.asTimestamp().getTime())
```

## Expected behavior

Users who require BSON fidelity should have a supported way to preserve both components:

```json
{
"time": 1710000000,
"increment": 7
}
```

Existing users might depend on the current Connect Timestamp representation, so changing the default representation could be incompatible.

## Suggested improvement

Add an opt-in BSON Timestamp handling mode, for example:

```properties
transforms.unwrap.bson.timestamp.handling.mode=connect
transforms.unwrap.bson.timestamp.handling.mode=struct
```

Possible semantics:

* `connect` preserves the existing behavior for compatibility
* `struct` emits both `time` and `increment`
* an optional Extended JSON string mode could also be considered

Add regression coverage proving that two values with the same `time` and different `increment` remain distinguishable in the lossless mode.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.