apache / apache/iceberg

Column statistics are not collected when the column name is escaped

Open
#11,950 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.7.1 (latest release)

### Query engine

None

### Please describe the bug 🐞

When an Iceberg has many columns, it calculates statistics only up to 100 columns (can be changed using config `write.metadata.metrics.max-inferred-column-defaults`).

When a column name is escaped, there is a bug that it tries to use the escaped column name when calculating the statistics while the metrics mode was saved using the original column name.

For example, I have a column `$event_time` which is escaped to `_x24event_time`.
When building the metrics mode, it uses the original name in [MetricsConfig.java:138](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/MetricsConfig.java#L138):
![image (5)](https://github.com/user-attachments/assets/f731fd1b-7efd-4a6a-ac05-be1a2427c545)

while when closing the parquet writer, it tries to get the metrics mode for `_x24event_time` in [MetricsUtil.java:118](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/MetricsUtil.java#L118):

![image (4)](https://github.com/user-attachments/assets/01d1fa48-e984-4d29-986b-8a35e1f0efcb)

when it doesn't find the metrics mode, it uses the default which is `none` so the stats are not collected.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.