[Python] Reading flat dataset with `partitioning="hive"` results in partition schema equal to dataset schema
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Example code:
```
% ls flat/
4dc78608-2739-4b67-942a-6eb39d39fe62.999.parquet
% python
>>> hive_ds = pyarrow.dataset.dataset("flat/", partitioning="hive", format="parquet")
>>> hive_ds.partitioning
>>> hive_ds.partitioning.schema
_index_: int64
label: list
child 0, item: string
score: list
child 0, item: float
>>> hive_ds.schema
_index_: int64
label: list
child 0, item: string
score: list
child 0, item: float
>>>
>>> flat_ds = pyarrow.dataset.dataset("flat/", format="parquet")
>>> flat_ds.schema
_index_: int64
label: list
child 0, item: string
score: list
child 0, item: float
>>> flat_ds.partitioning
>>> flat_ds.partitioning is None
True
```
Notice how when using `partitioning="hive"`, the partitioning schema is the same as the dataset schema. Since the data is not, in fact, partitioned, I would expect `hive_ds.partitioning` to be `None`.
PyArrow 10.0.1, Linux, same result with data on local filesystem and when retrieving from GCS.
### Component(s)
Python
Contributor guide
Research direction
Reproduce the example with the flat/ directory using pyarrow.dataset.dataset and compare partitioning with schema for the hive and default cases. Start at the Python dataset partitioning entry point; done means an unpartitioned flat dataset does not acquire a partitioning schema when partitioning="hive" is requested, while genuinely partitioned data remains supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100