apache / apache/iceberg-python
Upsert fails after adding in a new column, target field not found
- 主要語言
- Python
- 星號
- 1.1k
- 分支
- 581
- 平均合併
- 1 天 17 小時
- 30 天內合併 PR
- 78
描述
### Apache Iceberg version
0.10.0 (latest release)
### Please describe the bug 🐞
The upsert works perfectly fine until I needed to add a new field in the table.
Add the new column in the table
```python
# Add in a column to an existing table
from pyiceberg.types import TimestamptzType, TimestampType
table = catalog.load_table(table_identifier)
(
table.update_schema()
.add_column("created_at", TimestamptzType(), doc="UTC created time", required=False)
.commit()
)
print("New schema:", table.schema())
```
Upsert the records
```python
# Batch the records in 1000s
for rb in arrow_table_fixed.to_batches(max_chunksize=1000):
batch_tbl = pa.Table.from_batches([rb])
# Upsert the data into the Iceberg table
try:
upd = iceberg_table.upsert(batch_tbl)
print("Upserted data into the Iceberg table.")
print(upd)
except Exception as e:
print(f"An error occurred during upsert: {e}")
```
Error message saying that the target schema doesn't have the new column
```error
An error occurred during upsert: Target schema's field names are not matching the table's field names: ['cik_str', 'ticker', 'title', 'created_at'], ['cik_str', 'ticker', 'title']
```
Checked the target schema on Iceberg and the column is definitely there
```python
# Get the schema from the Iceberg table
iceberg_table = catalog.load_table(table_identifier)
# 2) Get the PyArrow schema directly from the Iceberg schema
arrow_schema = iceberg_table.schema().as_arrow()
print(arrow_schema.schema)
```
output
```
cik_str: large_string not null
-- field metadata --
PARQUET:field_id: '1'
ticker: large_string not null
-- field metadata --
PARQUET:field_id: '2'
title: large_string
-- field metadata --
PARQUET:field_id: '3'
created_at: timestamp[us, tz=UTC]
-- field metadata --
doc: 'UTC created time'
PARQUET:field_id: '5'
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
首先,使用 table.update_schema()、catalog.load_table() 和 iceberg_table.upsert(),並加入 created_at 欄位,重現所提供的序列。追蹤 upsert 目標 schema 驗證,該驗證會回報欄位名稱不相符。完成標準是:加入欄位後,upsert 成功且不再出現目標 schema 錯誤,同時保留所示範的 schema 行為。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python
- 領域
- data-engineering, databases
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 活躍
- 描述清晰度
- 基本清楚
- 新手友好度
- 55/100