airbytehq / airbytehq/airbyte

GCS Data Lake (Iceberg) - List<Objects> Source Column type Issue

Offen
#85,251 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
autoteam community needs-triage team/use
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

### Topic

_No response_

### Relevant information

**ISSUE WITH AIRBYTE ICEBERG TYPE:**

Schema inference for the GCS Data Lake (Iceberg) destination checks the field's type and converts a top-level "list of objects" field to list. This does not match what the connector's own docs which says: the docs say Array → String (a single stringified value), not list. This mismatch between the inferred schema and what's actually delivered at write time causes the sync to crash.

Airbyte Docs reference: [GCS Data Lake | Airbyte Docs](https://docs.airbyte.com/integrations/destinations/gcs-data-lake)
(table: "Array → String (JSON-serialized value)").

On Checking the Airbyte CDK Code:
https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/bulk/toolkits/load-iceberg-parquet/src/main/kotlin/io/airbyte/cdk/load/data/iceberg/parquet/AirbyteTypeToIcebergSchema.kt

Where it actually breaks: Before writing, there's a check: is the column's schema in the target table declared as string → False, as it's declared as a `list`.

The writer then tries to insert the value into the schema - but the actual value is a list of objects, hence we get: `java.lang.IllegalArgumentException: ObjectValue should be mapped to StructType`

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.