dask / dask/dask

Should Dask drop partition columns when writing to parquet with partition_by?

Open
#6,827 2 comments 0 reactions 0 assignees View on GitHub
dataframe io needs attention parquet
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

In Spark and I believe Pandas, when you partition by a field and write to parquet, it drops the partition column since the folder path now contains the field data.

When I write to parquet using the below code, the correct parquet folders are created (i.e. name=XXXX), but the parquet files also have the name field which is the column I partitioned on. How can I use to_parquet to also drop the partition_on column?

df.to_parquet(f"s3://data", engine="pyarrow", compression="snappy", partition_on=['name'])

Seems like the code on github is supposed to drop the partition column:
![image](https://user-images.githubusercontent.com/35536427/98756848-36f8e780-2380-11eb-8ee6-f533d055a8f2.png)

- Dask version: 2.20.0
- Python version: 3.8
- Operating System: Debian
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.