apache / apache/arrow

HivePartitioning.discover() returning unexpected columns when reading a non-partitioned dataset

Open
#37,944 0 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I get a lot of good use out of this pattern to automatically read & parse HivePartitions out of a directory structure:

dataset = ds.dataset(path, partitioning=ds.HivePartitioning.discover())
dataset.partitioning.schema # lists the directory-based partitions

It seems to work great when the dataset being read is, in fact, Hive-Partitioned. However, it gives an unexpected result when I run this code over a non-partitioned dataset.

**Minimum reproducible example:**

```py
import os
import pandas as pd
import pyarrow.dataset as ds

# Create a dataset: One file in one folder.
root ="/tmp/example_dataset/"
os.makedirs(root, exist_ok=True)
df = pd.DataFrame([['alice', 40], ['bob', 22], ['carlos', 50]], columns=['Name', 'Age'])
df.to_csv(os.path.join(root, "part-0.csv"), index=False)

# Exhibit the bug
dataset = ds.dataset(root, format="csv", partitioning=ds.HivePartitioning.discover())
if dataset.partitioning.schema.names != []:
raise AssertionError(f"Read unexpected HivePartitioning {dataset.partitioning.schema.names}")
```

Note you can also reproduce this with non-partitioned Parquet datasets (`format="parquet"`); not only CSVs.

**Expected behavior:** The example dataset does not use hive-partitioning, so I expect `dataset.partitioning.schema.names` to be the empty list.

**Actual behavior:** Instead, it is all the columns inside a single CSV fragment:

AssertionError: Read unexpected HivePartitioning ['Name', 'Age']

### Platform
Platform: Linux amd64
Pyarrow versions tested: 12.0.1 and 13.0.0

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.