apache / apache/iceberg-python

Possible read-after-write consistency issue with multiple schema migration steps in Iceberg tables on AWS Glue + S3

未关闭
#2,599 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

### Apache Iceberg version

Pyiceberg 0.10.0
Pyiceberg-core 0.6.0

### Please describe the bug 🐞

This may be a hard one to pin down but I noticed that multiple schema migration steps executed sequentially in the same `update_schema` context *sometimes* result in Exceptions like column name not found when using Iceberg tables on AWS Glue. An example:

```python
with table.update_schema() as update:
update.rename_column("some_column", "renamed_column")
update.move_first("renamed_column") # this sometimes fails with an error
# that renamed column doesn't exist
```

I have not noticed it with other back-ends like SQLite, leading me to believe it is a Glue issue specifically where a write may not yet be reflected by the time of the next operation.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 table.update_schema() 流程开始,在 AWS Glue 和 S3 上复现先执行 rename_column、再执行 move_first 的顺序示例,并与 SQLite 进行比较。跟踪第一次 schema 写入是否在下一次操作之前可见。完成标准是解释清楚该不一致性,并添加回归测试,或明确记录剩余的后端限制。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python
领域
cloud, databases
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。