Parquet silently discards decimal values if they do not fit in declared type
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
If I declare a schema with a DecimalType(7,2) and I try and write a value that will not fit, such as 123,456.00 and then read it back, I get a null value i.e. the data is silently discarded.
I have written up a full example to demonstrate this, here:
https://github.com/andygrove/spark-parquet-decimal-bug
**Environment**: Ubuntu 15.10
**Reporter**: [Andy Grove](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=andygrove) / @andygrove
**Note**: *This issue was originally created as [PARQUET-578](https://issues.apache.org/jira/browse/PARQUET-578). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file or test is named. Start by running the full example linked in the issue and compare writing and reading a value that exceeds DecimalType(7,2); done means the out-of-range decimal is handled explicitly rather than silently returned as null.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100