apache / apache/doris-flink-connector

[Feature](mongodb-cdc) Add date.timezone option for MongoDB to Doris time conversion

Open
#666 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
385
Forks
279
Avg merge
2d 2h
Merged PRs (30d)
6

Description

## Problem Summary

When using `mongodb-sync-database` to sync MongoDB data to Doris, BSON Date fields are always converted using a hardcoded timezone (`Asia/Shanghai` in `MongoDateConverter`). There is no way for users to configure the target timezone, and even if one were added, the timezone configuration would not take effect on TaskManagers due to the use of a static variable that resets after serializer deserialization.

## Root Causes

1. **Hardcoded timezone**: `MongoDateConverter` hardcodes `Asia/Shanghai`, making it impossible for users in other regions to control conversion.

2. **Debezium serializes BSON Date as ISO 8601 strings**: The Change Stream JSON represents dates as strings like `\"2024-06-04T08:03:37Z\"` instead of `{\"$date\": ...}` objects. The existing `formatSpecialFieldData()` only handles the object form, so these strings bypass conversion entirely.

3. **Static variable not serializable to TaskManager**: `MongoDateConverter.timeZone` is a static variable set on the Flink client side. After serializer serialization to TaskManagers, it resets to default `\"UTC\"`.

## Proposed Changes

1. Add new `date.timezone` ConfigOption to `MongoDBDatabaseSync` (default `UTC`).
2. Add ISO 8601 string parsing support in `MongoJsonDebeziumSchemaChange.formatSpecialFieldData()`.
3. Pass timezone through `JsonDebeziumChangeContext` (Serializable) to ensure it reaches TaskManagers.
4. Remove hardcoded `Asia/Shanghai` from `MongoDateConverter`.

## Checklist

1. Does it affect the original behavior: Yes (adds timezone conversion)
2. Has unit tests been added: No
3. Has document been added or modified: No Need
4. Does it need to update dependencies: No
5. Are there any changes that cannot be rolled back: No

Contributor guide

Open the contributing guide

Research direction

Start by locating MongoDBDatabaseSync, MongoJsonDebeziumSchemaChange.formatSpecialFieldData(), JsonDebeziumChangeContext, and MongoDateConverter. Trace how the date.timezone value would travel from configuration through serialization to TaskManagers, and inspect the existing BSON Date conversion and ISO 8601 handling. Done means the configured timezone is applied to both date representations on TaskManagers, with tests covering the new behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.