apache / apache/arrow

[Python] Individual RangeIndex not preserved with partitioning

Open
#37,699 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I am testing with 13.0.0 and 9.0.0. I am saving multiple dataframes separately with the following function

def save(table):
pq.write_to_dataset(
table=table,
root_path=root_path,
partition_cols=['date'],
basename_template='part-{i}.parquet',
use_threads=use_threads,
use_legacy_dataset=False,
existing_data_behavior='overwrite_or_ignore',
use_deprecated_int96_timestamps=False,
compression=compression,
**kwargs,
)

Each dataframe `df` has unique `date` column values and it's own RangeIndex range(len(df)). With 'overwrite_or_ignore' I am saving all dataframes to the same root_path.
When I load any individual partition, everything looks good. However, when reading the root_path folder as a ParquetDataset, returned dataframe `df_all` doesn't preserve the indices of its partitions and its index becomes RangeIndex(len(df_all)).

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.