apache / apache/iceberg-python

Enable stats collection for nested fields and use write.metadata.metrics.max-inferred-column-defaults to control stats growth

未关闭
#2,699 4 条评论 5 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

### Feature Request / Improvement

I recently discovered that full stats collection (i.e. `lower_bounds`/`upper_bounds`) is explicitly [disabled](https://github.com/apache/iceberg-python/blob/7acdb12f5239594e593f2822de75c7a4c669f52f/pyiceberg/io/pyarrow.py#L2220-L2223) in PyIceberg for nested (i.e. struct child) fields.

This change was made in [this PR](https://github.com/apache/iceberg/pull/7831) and specifically [this commit](https://github.com/apache/iceberg/pull/7831/commits/6f7bd987e50d4978aeae333d759e34422f7cfb78).

It seems that this change may have been made to limit the number of fields whose stats are collected when default-full stats collection is enabled. However, after discussion it seems that simply adding support for the `write.metadata.metrics.max-inferred-column-defaults` table [property](https://iceberg.apache.org/docs/nightly/configuration/#write-properties) would be the preferred way to control stats growth. If this is implemented, re-enabling stats collection for nested fields should be a non-issue.

Stats collection for nested struct fields is important for schemas like [GeoParquet](https://geoparquet.org/releases/v1.1.0/) which store important primitive fields (in this case, bounding box xmin, ymin, xmax, ymax) using structs.

See also [this slack thread](https://apache-iceberg.slack.com/archives/C029EE6HQ5D/p1762221916113919) for discussion.

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 pyiceberg/io/pyarrow.py 中禁用嵌套字段统计信息的链接行附近开始,然后跟踪 write.metadata.metrics.max-inferred-column-defaults 的处理方式。完成的标准是嵌套 struct 字段可以收集下界和上界,同时表属性控制推断列统计信息的增长;如果在附近找到现有的统计信息相关测试,请使用它们验证这两种行为。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
data-engineering
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
活跃
描述清晰度
基本清楚
新手友好度
66/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。