Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Debezium DATETIME type ingested as TIMESTAMP adding unwanted timezone semantics

Aberta
#184 1 comentário 0 reações 0 responsáveis Ver no GitHub
enhancement
Linguagem predominante
Java
Estrelas
37
Forks
45
Merge médio
19h 50min
PRs com merge (30d)
5

Descrição

## Problem Summary
When using Debezium as a data source with the BigQuery Sink connector, timezone-agnostic `DATETIME` data from the source database is being incorrectly mapped to BigQuery's UTC-fixed TIMESTAMP type.

## Details
### Expected Behavior
The connector should map Debezium's `io.debezium.time.Timestamp` and `io.debezium.time.MicroTimestamp` logical types to BigQuery's native `DATETIME` type. This preserves the timezone-agnostic "local" semantics of the source data.

### Actual Behavior
The connector currently maps these logical types to BigQuery's `TIMESTAMP` type. Since BigQuery `TIMESTAMP` is always stored as a UTC point-in-time, this forces an implicit UTC offset onto the data.

### Business Impact
This could result in a mismatch, where local time values appear shifted when displayed or queried in different timezones. Users would be unable to maintain local time representation from their source database within BigQuery.

### Potential Cause
In [DebeziumLogicalConverters.java](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/bc35e64fcde7cc3e227561a99102669289092fef/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/convert/logicaltype/DebeziumLogicalConverters.java), both the TimestampConverter and MicroTimestampConverter are explicitly hardcoded to use `LegacySQLTypeName.TIMESTAMP` ([1](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/bc35e64fcde7cc3e227561a99102669289092fef/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/convert/logicaltype/DebeziumLogicalConverters.java#L201-L205), [2](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/bc35e64fcde7cc3e227561a99102669289092fef/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/convert/logicaltype/DebeziumLogicalConverters.java#L148-L152)). There is currently no configuration option to override this mapping to use the `DATETIME` type.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.