Provide a lossless handling mode for MongoDB BSON Timestamp values
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
## Problem
`ExtractNewDocumentState` converts a MongoDB BSON Timestamp into a Kafka Connect Timestamp using only the BSON `time` component.
The BSON `increment` component is silently discarded.
As a result, distinct BSON values such as `Timestamp(1710000000, 7)` and `Timestamp(1710000000, 8)` produce the same transformed value.
This prevents consumers from reconstructing the original BSON value and can collapse values that are distinct in MongoDB.
## Environment
* Percona Server for MongoDB 8.0.29-13
* Debezium MongoDB Source 3.7.0-SNAPSHOT
* Kafka 3.8.0
The same behavior was reproduced with Debezium 3.6.1.Final.
## Reproduction
Configure the connector with `ExtractNewDocumentState` and insert:
```javascript
db.timestamp_values.insertMany([
{
_id: "timestamp-1",
value: Timestamp(1710000000, 7)
},
{
_id: "timestamp-2",
value: Timestamp(1710000000, 8)
}
]);
```
## Actual behavior
Both BSON values are converted to the same Connect Timestamp representing:
```text
2024-03-09T16:00:00Z
```
The increment values `7` and `8` are not present in the schema or payload.
The conversion currently behaves equivalently to:
```java
new Date(1000L * value.asTimestamp().getTime())
```
## Expected behavior
Users who require BSON fidelity should have a supported way to preserve both components:
```json
{
"time": 1710000000,
"increment": 7
}
```
Existing users might depend on the current Connect Timestamp representation, so changing the default representation could be incompatible.
## Suggested improvement
Add an opt-in BSON Timestamp handling mode, for example:
```properties
transforms.unwrap.bson.timestamp.handling.mode=connect
transforms.unwrap.bson.timestamp.handling.mode=struct
```
Possible semantics:
* `connect` preserves the existing behavior for compatibility
* `struct` emits both `time` and `increment`
* an optional Extended JSON string mode could also be considered
Add regression coverage proving that two values with the same `time` and different `increment` remain distinguishable in the lossless mode.
Contributor guide
Assessment
This issue has not been assessed yet.