Statistic and Filter need a mechanism to get customized comparator from high layer user
- Lenguaje dominante
- Java
- Estrellas
- 3.1k
- Forks
- 1.6k
- Merge medio
- 3 d 12 h
- PR fusionados (30 d)
- 33
Descripción
As discussed in HIVE-10254, we might need a customized comparator from high layer user for generating statistic when writing and applying filter when reading.
The problem is that (use Decimal type in Hive as an example):
Decimal in Hive is mapped to Binary in Parquet. When using predicate and statistic to filter values, comparing Binary values in Parquet cannot reflect the correct relationship of Decimal values in Hive. This type mapping causes 2 problems:
1. When writing Decimal column, Binary.compareTo() is used to judge and set the column statistic (min, max). The generated statistic value is not correct from a Decimal perspective.
2. When reading with Predicate (also Filter), in which the expected Decimal value is converted to Binary type, Binary.compareTo() is used to compare the expected value and column statistic value. They are Binary perspective, and also the result is not right.
We could add an interface for customized comparator, and high level user like Hive provides the comparator to Parquet, since Hive knows how to decode the binary to Decimal and compare. Then Parquet could switch between customized and original comparison method.
**Reporter**: [Dong Chen](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=dongc) / @dongc
**Assignee**: [Dong Chen](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=dongc) / @dongc
#### Related issues:
- [Parquet PPD support DECIMAL](https://issues.apache.org/jira/browse/HIVE-10254) (blocks)
**Note**: *This issue was originally created as [PARQUET-281](https://issues.apache.org/jira/browse/PARQUET-281). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
Comienza siguiendo la ruta de generación de estadísticas para escrituras y la ruta de comparación de predicados/filtros para lecturas descritas en el issue, usando HIVE-10254 para el comportamiento de Decimal. Determina dónde podría proporcionarse un comparador de alto nivel sin cambiar el comportamiento de comparación existente. Se considera terminado cuando la comparación personalizada ordena correctamente los valores Decimal tanto en las estadísticas como en el filtrado.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- java
- Área
- data-engineering
- Tipo de issue
- Nueva funcionalidad
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Estado de actividad
- Estancado
- Claridad
- Bastante claro
- Aptitud para principiantes
- 25/100