apache / apache/parquet-java

Statistic and Filter need a mechanism to get customized comparator from high layer user

未关闭
#1,810 6 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Parquet Priority: Major Type: enhancement
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

As discussed in HIVE-10254, we might need a customized comparator from high layer user for generating statistic when writing and applying filter when reading.

The problem is that (use Decimal type in Hive as an example):
Decimal in Hive is mapped to Binary in Parquet. When using predicate and statistic to filter values, comparing Binary values in Parquet cannot reflect the correct relationship of Decimal values in Hive. This type mapping causes 2 problems:
1. When writing Decimal column, Binary.compareTo() is used to judge and set the column statistic (min, max). The generated statistic value is not correct from a Decimal perspective.
2. When reading with Predicate (also Filter), in which the expected Decimal value is converted to Binary type, Binary.compareTo() is used to compare the expected value and column statistic value. They are Binary perspective, and also the result is not right.

We could add an interface for customized comparator, and high level user like Hive provides the comparator to Parquet, since Hive knows how to decode the binary to Decimal and compare. Then Parquet could switch between customized and original comparison method.

**Reporter**: [Dong Chen](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=dongc) / @dongc
**Assignee**: [Dong Chen](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=dongc) / @dongc
#### Related issues:
- [Parquet PPD support DECIMAL](https://issues.apache.org/jira/browse/HIVE-10254) (blocks)

**Note**: *This issue was originally created as [PARQUET-281](https://issues.apache.org/jira/browse/PARQUET-281). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先,按照 issue 中的描述,跟踪写入的统计信息生成路径以及读取的谓词/过滤器比较路径,并使用 HIVE-10254 了解 Decimal 的行为。确定可以在何处提供高级比较器,同时不改变现有的比较行为。当自定义比较能够在统计信息和过滤过程中都正确排序 Decimal 值时,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。