apache / apache/arrow

Error message regarding invalid values (i.e Rescaling Decimal128 value would cause data loss) does not show the offending values

Open
#44,061 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Converting from decimal.Decimal to PyAarrows Decimal has given me the
`pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss` error.

I understand that to fix this, I have to ensure that the DecimalType (I am using PySpark, and their schema) I have to max the precision and scaling to the value I want to store.

However, to not allocate more memory than needed, I want to right-size the data; however, receiving the above error in production, without any indication of which value is outside this range, is frustrating and not very helpful.

There are similar lacks in many other error messages regarding invalid data

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.