Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka
Debezium DATETIME type ingested as TIMESTAMP adding unwanted timezone semantics
- 主要語言
- Java
- 星號
- 37
- 分支
- 45
- 平均合併
- 19 小時 50 分鐘
- 30 天內合併 PR
- 5
描述
## Problem Summary
When using Debezium as a data source with the BigQuery Sink connector, timezone-agnostic `DATETIME` data from the source database is being incorrectly mapped to BigQuery's UTC-fixed TIMESTAMP type.
## Details
### Expected Behavior
The connector should map Debezium's `io.debezium.time.Timestamp` and `io.debezium.time.MicroTimestamp` logical types to BigQuery's native `DATETIME` type. This preserves the timezone-agnostic "local" semantics of the source data.
### Actual Behavior
The connector currently maps these logical types to BigQuery's `TIMESTAMP` type. Since BigQuery `TIMESTAMP` is always stored as a UTC point-in-time, this forces an implicit UTC offset onto the data.
### Business Impact
This could result in a mismatch, where local time values appear shifted when displayed or queried in different timezones. Users would be unable to maintain local time representation from their source database within BigQuery.
### Potential Cause
In [DebeziumLogicalConverters.java](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/bc35e64fcde7cc3e227561a99102669289092fef/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/convert/logicaltype/DebeziumLogicalConverters.java), both the TimestampConverter and MicroTimestampConverter are explicitly hardcoded to use `LegacySQLTypeName.TIMESTAMP` ([1](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/bc35e64fcde7cc3e227561a99102669289092fef/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/convert/logicaltype/DebeziumLogicalConverters.java#L201-L205), [2](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/bc35e64fcde7cc3e227561a99102669289092fef/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/convert/logicaltype/DebeziumLogicalConverters.java#L148-L152)). There is currently no configuration option to override this mapping to use the `DATETIME` type.
貢獻指南
這個儲存庫沒有索引到貢獻指南
評估
這個 Issue 還沒有評估資料。