apache / apache/arrow

[Python][C++] a float32 value in a structure is stored incorrectly above a given number

Open
#37,956 8 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I am trying to write a structured data using the following schema.
Using pyarrow==13.0.0
on a MacOS Monterey 12.1, M1 macbook

It seems like, that above a certain value, a floating point number is written incorrectly to the resulting parquet file. The error seems to depend on the actual value of the number.

This is the minimum code where I could replicate this behaviour:
```python
import pandas as pd
import pyarrow as pa

schema = pa.schema(
[
pa.field(
name="root",
type=pa.struct(
[
pa.field(name="timestamp", type=pa.float32(), nullable=True),
]
),
nullable=True,
)
]
)

def test_pyarrow_write_bug():

data = [
{'root': {'timestamp': 1.0}},
{'root': {'timestamp': 1693260000.0}},
{'root': {'timestamp': 846630000.0}},
{'root': {'timestamp': 423315000.0}},
{'root': {'timestamp': 211657500.0}},
{'root': {'timestamp': 105828750.0}},
{'root': {'timestamp': 52914375.0}},
{'root': {'timestamp': 26457187.0}},
]
df = pd.DataFrame(data)
table = pa.Table.from_pandas(df, schema=schema)
pa.parquet.write_table(table, 'test_pyarrow_write_bug.parquet')

df_read = pd.read_parquet('test_pyarrow_write_bug.parquet')
assert df['root'].to_list() == df_read['root'].to_list()
```
image

I would expect the written data and read file contents to be equal

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.