Statistics is not available for DECIMAL types
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
According to parquet format specification columns annotated as DECIMAL should use SIGNED comparator and statistics should be available. The sort order returned by `org.apache.parquet.format.converter.ParquetMetadataConverter` for DECIMAL is `SortOrder.UNKNOWN` which contradicts the specification and makes statistics for DECIMAL types unavailable.
**Reporter**: [Vlad Rozov](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vrozov) / @vrozov
**Assignee**: [Vlad Rozov](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vrozov) / @vrozov
#### Related issues:
- [Allow for Unsigned Statistics in Binary Type](https://github.com/apache/parquet-format/issues/314) (is caused by)
#### PRs and other links:
- [GitHub Pull Request #494](https://github.com/apache/parquet-mr/pull/494)
- [Decimal format specification](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal)
**Note**: *This issue was originally created as [PARQUET-1322](https://issues.apache.org/jira/browse/PARQUET-1322). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in org.apache.parquet.format.converter.ParquetMetadataConverter and inspect how DECIMAL sort order is determined. Compare that behavior with the Decimal format specification and the related statistics discussion; done means DECIMAL columns use the specified SIGNED comparator and their statistics are available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100