Aiven-Open / Aiven-Open/bigquery-connector-for-apache-kafka

Debezium JSON type ingested as STRING instead of native BigQuery JSON

Offen
#183 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement
Vorherrschende Sprache
Java
Sterne
37
Forks
45
Ø Merge
19 Std. 50 Min.
Gemergte PRs (30 T.)
5

Beschreibung

## Problem Summary

When using Debezium as a data source with the BigQuery Sink connector, JSON data from the source database is not being correctly mapped to BigQuery's native JSON type.

## Details

### Expected Behavior
The connector should detect the `io.debezium.data.Json` metadata and create a column with the BigQuery JSON type.

### Actual Behavior
The connector currently ingests this data as a plain STRING.

### Business Impact
This results in a loss of type-specific functionality within BigQuery. More specifically, users are unable to leverage native BigQuery JSON features, such as JSON path extraction and specialized SQL functions, on the ingested columns.

If possible, changes should be guarded behind a config value to ensure that existing workloads aren’t affected.

### Potential Cause
There is no JSON converter registered in [DebeziumLogicalConverter](https://github.com/Aiven-Open/bigquery-connector-for-apache-kafka/blob/bc35e64fcde7cc3e227561a99102669289092fef/kcbq-connector/src/main/java/com/wepay/kafka/connect/bigquery/convert/logicaltype/DebeziumLogicalConverters.java), which seems to result in the connector treating the data object as a String.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.