apache / apache/arrow

[Python] pyarrow large integer conversion

Open
#32,664 2 comments 0 reactions 0 assignees View on GitHub
Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

I have a json document that looks like this. 

{"number": 123451234512345123451234512}

I then run the below code. 

>>> from pyarrow.json import read_json
>>> pyarrow_table = read_json('pyarrow_test.json')
>>> pyarrow_table['number'][0].as_py().as_integer_ratio()
(123451234512345125900779520, 1)

notice the float that I get looks like it has been rounded or modified in some way.

 

Am I reading this file incorrectly or is there an issue with the conversion of this number?

**Reporter**: [Donald Freeman](https://issues.apache.org/jira/browse/ARROW-17393)

**Note**: *This issue was originally created as [ARROW-17393](https://issues.apache.org/jira/browse/ARROW-17393). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the example with pyarrow.json.read_json and the supplied large-integer JSON value. Trace how the Python reader represents the number and determine whether the observed rounding is expected; done means the behavior is corrected or clearly covered by an agreed regression test or explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.