GCS Data Lake (Iceberg) - List<Objects> Source Column type Issue
- 主要語言
- Python
- 星號
- 22.1k
- 分支
- 5.4k
- 平均合併
- 5 小時
- 30 天內合併 PR
- 671
描述
### Topic
_No response_
### Relevant information
**ISSUE WITH AIRBYTE ICEBERG TYPE:**
Schema inference for the GCS Data Lake (Iceberg) destination checks the field's type and converts a top-level "list of objects" field to list. This does not match what the connector's own docs which says: the docs say Array → String (a single stringified value), not list. This mismatch between the inferred schema and what's actually delivered at write time causes the sync to crash.
Airbyte Docs reference: [GCS Data Lake | Airbyte Docs](https://docs.airbyte.com/integrations/destinations/gcs-data-lake)
(table: "Array → String (JSON-serialized value)").
On Checking the Airbyte CDK Code:
https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/bulk/toolkits/load-iceberg-parquet/src/main/kotlin/io/airbyte/cdk/load/data/iceberg/parquet/AirbyteTypeToIcebergSchema.kt
Where it actually breaks: Before writing, there's a check: is the column's schema in the target table declared as string → False, as it's declared as a `list`.
The writer then tries to insert the value into the schema - but the actual value is a list of objects, hence we get: `java.lang.IllegalArgumentException: ObjectValue should be mapped to StructType`
貢獻指南
研究方向
Start with airbyte-cdk/bulk/toolkits/load-iceberg-parquet/src/main/kotlin/io/airbyte/cdk/load/data/iceberg/parquet/AirbyteTypeToIcebergSchema.kt and compare its top-level list-of-objects inference with the GCS Data Lake documentation, where Array is mapped to a JSON-serialized String. Trace the inferred schema into the writer check. Done means the inferred target schema matches the value delivered at write time and the sync no longer raises the reported IllegalArgumentException.
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- google-cloud, kotlin
- 領域
- data-engineering, databases
- Issue 類型
- 缺陷
- 難度
- 3/5
- 預估耗時
- 1-2 天
- 活躍度
- 活躍
- 描述清晰度
- 描述清楚
- 新手友好度
- 70/100