apache / apache/iceberg-python
Possible read-after-write consistency issue with multiple schema migration steps in Iceberg tables on AWS Glue + S3
- 主要语言
- Python
- 星标
- 1.1k
- 派生
- 581
- 平均合并
- 1 天 17 小时
- 30 天内合并 PR
- 78
描述
### Apache Iceberg version
Pyiceberg 0.10.0
Pyiceberg-core 0.6.0
### Please describe the bug 🐞
This may be a hard one to pin down but I noticed that multiple schema migration steps executed sequentially in the same `update_schema` context *sometimes* result in Exceptions like column name not found when using Iceberg tables on AWS Glue. An example:
```python
with table.update_schema() as update:
update.rename_column("some_column", "renamed_column")
update.move_first("renamed_column") # this sometimes fails with an error
# that renamed column doesn't exist
```
I have not noticed it with other back-ends like SQLite, leading me to believe it is a Glue issue specifically where a write may not yet be reflected by the time of the next operation.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
贡献指南
这个仓库没有索引到贡献指南
调研方向
从 table.update_schema() 流程开始,在 AWS Glue 和 S3 上复现先执行 rename_column、再执行 move_first 的顺序示例,并与 SQLite 进行比较。跟踪第一次 schema 写入是否在下一次操作之前可见。完成标准是解释清楚该不一致性,并添加回归测试,或明确记录剩余的后端限制。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- aws, python
- 领域
- cloud, databases
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 45/100