apache / apache/arrow

[Python] Float value keys written as `x` instead of `x.0` when `x` has no remainder during Hive partitioning

Open
#39,763 1 comment 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

This may be expected behavior, but when hive-partioning a dataset, if a float column is used as the partition key and the values are all integer-like (e.g. `x.0`) then the output path is `value=x` instead of `value=x.0` and cannot be distinguished from an integer column when subsequently re-read.

### Reproducible Example

```python
from pathlib import Path
import pyarrow as pa
import pyarrow.parquet as pq

table_out = pa.table(
{
"float": [1.0, 2.0, 3.0],
"int": [1, 2, 3],
}
)

pq.write_to_dataset(
table_out ,
root_path=Path() / "test_parquet",
partition_cols=["float"],
)
```

### Output:
```shell
$ ls test_parquet
'float=1' 'float=2' 'float=3'
```

### Possibly Improved Expected Behavior:
```shell
$ ls test_parquet
'float=1.0' 'float=2.0' 'float=3.0'
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.