apache / apache/iceberg-python

Upsert fails after adding in a new column, target field not found

未關閉
#2,467 6 則留言 6 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
1.1k
分支
581
平均合併
1 天 17 小時
30 天內合併 PR
78

描述

### Apache Iceberg version

0.10.0 (latest release)

### Please describe the bug 🐞

The upsert works perfectly fine until I needed to add a new field in the table.

Add the new column in the table
```python
# Add in a column to an existing table

from pyiceberg.types import TimestamptzType, TimestampType

table = catalog.load_table(table_identifier)

(
table.update_schema()
.add_column("created_at", TimestamptzType(), doc="UTC created time", required=False)
.commit()
)

print("New schema:", table.schema())
```

Upsert the records
```python
# Batch the records in 1000s
for rb in arrow_table_fixed.to_batches(max_chunksize=1000):
batch_tbl = pa.Table.from_batches([rb])

# Upsert the data into the Iceberg table
try:
upd = iceberg_table.upsert(batch_tbl)
print("Upserted data into the Iceberg table.")
print(upd)
except Exception as e:
print(f"An error occurred during upsert: {e}")
```

Error message saying that the target schema doesn't have the new column
```error
An error occurred during upsert: Target schema's field names are not matching the table's field names: ['cik_str', 'ticker', 'title', 'created_at'], ['cik_str', 'ticker', 'title']
```

Checked the target schema on Iceberg and the column is definitely there
```python
# Get the schema from the Iceberg table
iceberg_table = catalog.load_table(table_identifier)
# 2) Get the PyArrow schema directly from the Iceberg schema
arrow_schema = iceberg_table.schema().as_arrow()
print(arrow_schema.schema)
```

output
```
cik_str: large_string not null
-- field metadata --
PARQUET:field_id: '1'
ticker: large_string not null
-- field metadata --
PARQUET:field_id: '2'
title: large_string
-- field metadata --
PARQUET:field_id: '3'
created_at: timestamp[us, tz=UTC]
-- field metadata --
doc: 'UTC created time'
PARQUET:field_id: '5'
```

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

首先,使用 table.update_schema()、catalog.load_table() 和 iceberg_table.upsert(),並加入 created_at 欄位,重現所提供的序列。追蹤 upsert 目標 schema 驗證,該驗證會回報欄位名稱不相符。完成標準是:加入欄位後,upsert 成功且不再出現目標 schema 錯誤,同時保留所示範的 schema 行為。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
data-engineering, databases
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
活躍
描述清晰度
基本清楚
新手友好度
55/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。