airbytehq / airbytehq/airbyte

GCS Data Lake (Iceberg) - List<Objects> Source Column type Issue

未關閉
#85,251 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
autoteam community needs-triage team/use
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Topic

_No response_

### Relevant information

**ISSUE WITH AIRBYTE ICEBERG TYPE:**

Schema inference for the GCS Data Lake (Iceberg) destination checks the field's type and converts a top-level "list of objects" field to list. This does not match what the connector's own docs which says: the docs say Array → String (a single stringified value), not list. This mismatch between the inferred schema and what's actually delivered at write time causes the sync to crash.

Airbyte Docs reference: [GCS Data Lake | Airbyte Docs](https://docs.airbyte.com/integrations/destinations/gcs-data-lake)
(table: "Array → String (JSON-serialized value)").

On Checking the Airbyte CDK Code:
https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/bulk/toolkits/load-iceberg-parquet/src/main/kotlin/io/airbyte/cdk/load/data/iceberg/parquet/AirbyteTypeToIcebergSchema.kt

Where it actually breaks: Before writing, there's a check: is the column's schema in the target table declared as string → False, as it's declared as a `list`.

The writer then tries to insert the value into the schema - but the actual value is a list of objects, hence we get: `java.lang.IllegalArgumentException: ObjectValue should be mapped to StructType`

貢獻指南

開啟貢獻指南

研究方向

Start with airbyte-cdk/bulk/toolkits/load-iceberg-parquet/src/main/kotlin/io/airbyte/cdk/load/data/iceberg/parquet/AirbyteTypeToIcebergSchema.kt and compare its top-level list-of-objects inference with the GCS Data Lake documentation, where Array is mapped to a JSON-serialized String. Trace the inferred schema into the writer check. Done means the inferred target schema matches the value delivered at write time and the sync no longer raises the reported IllegalArgumentException.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
google-cloud, kotlin
領域
data-engineering, databases
Issue 類型
缺陷
難度
3/5
預估耗時
1-2 天
活躍度
活躍
描述清晰度
描述清楚
新手友好度
70/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。