HivePartitioning.discover() returning unexpected columns when reading a non-partitioned dataset
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I get a lot of good use out of this pattern to automatically read & parse HivePartitions out of a directory structure:
dataset = ds.dataset(path, partitioning=ds.HivePartitioning.discover())
dataset.partitioning.schema # lists the directory-based partitions
It seems to work great when the dataset being read is, in fact, Hive-Partitioned. However, it gives an unexpected result when I run this code over a non-partitioned dataset.
**Minimum reproducible example:**
```py
import os
import pandas as pd
import pyarrow.dataset as ds
# Create a dataset: One file in one folder.
root ="/tmp/example_dataset/"
os.makedirs(root, exist_ok=True)
df = pd.DataFrame([['alice', 40], ['bob', 22], ['carlos', 50]], columns=['Name', 'Age'])
df.to_csv(os.path.join(root, "part-0.csv"), index=False)
# Exhibit the bug
dataset = ds.dataset(root, format="csv", partitioning=ds.HivePartitioning.discover())
if dataset.partitioning.schema.names != []:
raise AssertionError(f"Read unexpected HivePartitioning {dataset.partitioning.schema.names}")
```
Note you can also reproduce this with non-partitioned Parquet datasets (`format="parquet"`); not only CSVs.
**Expected behavior:** The example dataset does not use hive-partitioning, so I expect `dataset.partitioning.schema.names` to be the empty list.
**Actual behavior:** Instead, it is all the columns inside a single CSV fragment:
AssertionError: Read unexpected HivePartitioning ['Name', 'Age']
### Platform
Platform: Linux amd64
Pyarrow versions tested: 12.0.1 and 13.0.0
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.