apache / apache/arrow

[Python] Reading flat dataset with `partitioning="hive"` results in partition schema equal to dataset schema

Open
#33,668 4 comments 0 reactions 0 assignees View on GitHub
Component: Python Priority: Critical Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Example code:
```
% ls flat/
4dc78608-2739-4b67-942a-6eb39d39fe62.999.parquet
% python
>>> hive_ds = pyarrow.dataset.dataset("flat/", partitioning="hive", format="parquet")
>>> hive_ds.partitioning

>>> hive_ds.partitioning.schema
_index_: int64
label: list
child 0, item: string
score: list
child 0, item: float
>>> hive_ds.schema
_index_: int64
label: list
child 0, item: string
score: list
child 0, item: float
>>>
>>> flat_ds = pyarrow.dataset.dataset("flat/", format="parquet")
>>> flat_ds.schema
_index_: int64
label: list
child 0, item: string
score: list
child 0, item: float
>>> flat_ds.partitioning
>>> flat_ds.partitioning is None
True
```

Notice how when using `partitioning="hive"`, the partitioning schema is the same as the dataset schema. Since the data is not, in fact, partitioned, I would expect `hive_ds.partitioning` to be `None`.

PyArrow 10.0.1, Linux, same result with data on local filesystem and when retrieving from GCS.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Reproduce the example with the flat/ directory using pyarrow.dataset.dataset and compare partitioning with schema for the hive and default cases. Start at the Python dataset partitioning entry point; done means an unpartitioned flat dataset does not acquire a partitioning schema when partitioning="hive" is requested, while genuinely partitioned data remains supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.