GCS Data Lake (Iceberg) - List<Objects> Source Column type Issue
- Vorherrschende Sprache
- Python
- Sterne
- 22.1k
- Forks
- 5.3k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
### Topic
_No response_
### Relevant information
**ISSUE WITH AIRBYTE ICEBERG TYPE:**
Schema inference for the GCS Data Lake (Iceberg) destination checks the field's type and converts a top-level "list of objects" field to list. This does not match what the connector's own docs which says: the docs say Array → String (a single stringified value), not list. This mismatch between the inferred schema and what's actually delivered at write time causes the sync to crash.
Airbyte Docs reference: [GCS Data Lake | Airbyte Docs](https://docs.airbyte.com/integrations/destinations/gcs-data-lake)
(table: "Array → String (JSON-serialized value)").
On Checking the Airbyte CDK Code:
https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/bulk/toolkits/load-iceberg-parquet/src/main/kotlin/io/airbyte/cdk/load/data/iceberg/parquet/AirbyteTypeToIcebergSchema.kt
Where it actually breaks: Before writing, there's a check: is the column's schema in the target table declared as string → False, as it's declared as a `list`.
The writer then tries to insert the value into the schema - but the actual value is a list of objects, hence we get: `java.lang.IllegalArgumentException: ObjectValue should be mapped to StructType`
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.