dask / dask/dask

Provide `DataFrame.to_pickle`

Open
#9,384 7 comments 0 reactions 0 assignees View on GitHub
dataframe feature needs attention
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

Currently the best supported way to store the distributed dataframe to disk is using `to_parquet`. This is great, but there are some use cases where pickle would work better, for example when the dataframe contains native Python types that aren't easily represented in parquet. Also, the type inference for parquet isn't perfect, so often an `object` dtype that is e.g. clearly a `list[str]` gets inferred as a string and serialization fails. Pickle serialization ideally would Just Work™ in a way that parquet doesn't necessarily.

The implementation I imagine would be fairly easy. Each partition is a dataframe, and so we can just `df.to_pickle` each of these into a separate pickle file. In fact, it seems like other output formats that pandas already supports could be supported in dask in the same way. Is there some difficult part of this that I'm missing? I would be happy to work on this feature if it's feasible for someone like me who isn't intimately familiar with the library internals.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.