apache / apache/iceberg-python
Upsert with None values fails on "Invalid literal value: None"
- 主要语言
- Python
- 星标
- 1.1k
- 派生
- 581
- 平均合并
- 1 天 17 小时
- 30 天内合并 PR
- 78
描述
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
Upserting a table fails when the input dataframe contains `None` in a join column.
I've reproduced this error by editing `test_upsert_with_nulls` from #1861, adding this at the end:
```python
# upsert table with null value
data_with_null = pa.Table.from_pylist(
[
{"foo": None, "bar": 1, "baz": False},
],
schema=schema,
)
upd = table.upsert(data_with_null, join_cols=["foo"])
```
The `foo` column contains `None`, causing `TypeError: Invalid literal value: None`.
```
tests/table/test_upsert.py:720: in test_upsert_with_nulls
upd = table.upsert(data_with_null, join_cols=["foo"])
pyiceberg/table/__init__.py:1343: in upsert
return tx.upsert(
pyiceberg/table/__init__.py:798: in upsert
matched_predicate = upsert_util.create_match_filter(df, join_cols)
pyiceberg/table/upsert_util.py:37: in create_match_filter
return In(join_cols[0], unique_keys[0].to_pylist())
pyiceberg/expressions/__init__.py:682: in __new__
literals_set: Set[Literal[L]] = _to_literal_set(literals)
pyiceberg/expressions/__init__.py:52: in _to_literal_set
return {_to_literal(v) for v in values}
pyiceberg/expressions/__init__.py:52: in
return {_to_literal(v) for v in values}
pyiceberg/expressions/__init__.py:59: in _to_literal
return literal(value)
pyiceberg/expressions/literals.py:159: in literal
raise TypeError(f"Invalid literal value: {repr(value)}")
E TypeError: Invalid literal value: None
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
贡献指南
这个仓库没有索引到贡献指南
调研方向
从 tests/table/test_upsert.py 开始,尤其关注 test_upsert_with_nulls 和新增的复现用例,然后阅读 pyiceberg/table/upsert_util.py:create_match_filter 以及 traceback 中展示的表达式字面量处理。与社区确认 join 值为 null 时的预期行为,然后让 upsert 场景正常工作,并添加或更新一个能够通过的回归测试。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- databases
- Issue 类型
- 缺陷
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 活跃度
- 活跃
- 描述清晰度
- 基本清楚
- 新手友好度
- 68/100