dask / dask/partd

Support for pandas DataFrame subclasses

Open
#52 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
107
Forks
32
PR merge metrics
No merged PRs in 30d

Description

When dask uses partd for eg shuffle operations, the dataframes always come back as a `pandas.DataFrame`, even if a subclass was stored (xref https://github.com/geopandas/dask-geopandas/issues/59#issuecomment-864469674).

For example:

```
import geopandas
gdf = geopandas.read_file(geopandas.datasets.get_path("naturalearth_lowres"))

import partd
# dask.dataframe shuffle operations use PandasBlocks
p = partd.PandasBlocks(partd.Dict())

p.append({"gdf": gdf})
res = p.get("gdf")

>>> type(gdf)
pandas.core.frame.DataFrame
>>> type(res)
pandas.core.frame.DataFrame
```

To be able to use dask's shuffle operations with `dask_geopandas`, which uses a pandas subclass as the partition type, the subclass should be preserved in the partd roundtrip (or are there other ways that you can override / dispatch this operation in dask?).
I was wondering how other dask.dataframe subclasses handle this, but eg `dask_cudf` doesn't seem to support "disk"-based shuffling.

Contributor guide

Open the contributing guide

Research direction

Start at the partd.PandasBlocks(partd.Dict()) entry point shown in the example and trace its append/get roundtrip. Done means a stored pandas DataFrame subclass, such as the GeoPandas example, is returned with its subclass preserved and the dask shuffle use case works.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.