Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Debezium DATETIME type ingested as TIMESTAMP adding unwanted timezone semantics

Ouverte
#184 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub
enhancement
Langage dominant
Java
Étoiles
37
Forks
45
Merge moyen
19 h 50 min
PR mergées (30 j)
5

Description

## Problem Summary
When using Debezium as a data source with the BigQuery Sink connector, timezone-agnostic `DATETIME` data from the source database is being incorrectly mapped to BigQuery's UTC-fixed TIMESTAMP type.

## Details
### Expected Behavior
The connector should map Debezium's `io.debezium.time.Timestamp` and `io.debezium.time.MicroTimestamp` logical types to BigQuery's native `DATETIME` type. This preserves the timezone-agnostic "local" semantics of the source data.

### Actual Behavior
The connector currently maps these logical types to BigQuery's `TIMESTAMP` type. Since BigQuery `TIMESTAMP` is always stored as a UTC point-in-time, this forces an implicit UTC offset onto the data.

### Business Impact
This could result in a mismatch, where local time values appear shifted when displayed or queried in different timezones. Users would be unable to maintain local time representation from their source database within BigQuery.

### Potential Cause
In [DebeziumLogicalConverters.java](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/bc35e64fcde7cc3e227561a99102669289092fef/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/convert/logicaltype/DebeziumLogicalConverters.java), both the TimestampConverter and MicroTimestampConverter are explicitly hardcoded to use `LegacySQLTypeName.TIMESTAMP` ([1](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/bc35e64fcde7cc3e227561a99102669289092fef/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/convert/logicaltype/DebeziumLogicalConverters.java#L201-L205), [2](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/bc35e64fcde7cc3e227561a99102669289092fef/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/convert/logicaltype/DebeziumLogicalConverters.java#L148-L152)). There is currently no configuration option to override this mapping to use the `DATETIME` type.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.