Should Dask drop partition columns when writing to parquet with partition_by?
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
In Spark and I believe Pandas, when you partition by a field and write to parquet, it drops the partition column since the folder path now contains the field data.
When I write to parquet using the below code, the correct parquet folders are created (i.e. name=XXXX), but the parquet files also have the name field which is the column I partitioned on. How can I use to_parquet to also drop the partition_on column?
df.to_parquet(f"s3://data", engine="pyarrow", compression="snappy", partition_on=['name'])
Seems like the code on github is supposed to drop the partition column:

- Dask version: 2.20.0
- Python version: 3.8
- Operating System: Debian
- Install method (conda, pip, source): conda
Contributor guide
Assessment
This issue has not been assessed yet.