Provide `DataFrame.to_pickle`
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
Currently the best supported way to store the distributed dataframe to disk is using `to_parquet`. This is great, but there are some use cases where pickle would work better, for example when the dataframe contains native Python types that aren't easily represented in parquet. Also, the type inference for parquet isn't perfect, so often an `object` dtype that is e.g. clearly a `list[str]` gets inferred as a string and serialization fails. Pickle serialization ideally would Just Work™ in a way that parquet doesn't necessarily.
The implementation I imagine would be fairly easy. Each partition is a dataframe, and so we can just `df.to_pickle` each of these into a separate pickle file. In fact, it seems like other output formats that pandas already supports could be supported in dask in the same way. Is there some difficult part of this that I'm missing? I would be happy to work on this feature if it's feasible for someone like me who isn't intimately familiar with the library internals.
Contributor guide
Assessment
This issue has not been assessed yet.