Support for pandas DataFrame subclasses
- Dominant language
- Python
- Stars
- 107
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
When dask uses partd for eg shuffle operations, the dataframes always come back as a `pandas.DataFrame`, even if a subclass was stored (xref https://github.com/geopandas/dask-geopandas/issues/59#issuecomment-864469674).
For example:
```
import geopandas
gdf = geopandas.read_file(geopandas.datasets.get_path("naturalearth_lowres"))
import partd
# dask.dataframe shuffle operations use PandasBlocks
p = partd.PandasBlocks(partd.Dict())
p.append({"gdf": gdf})
res = p.get("gdf")
>>> type(gdf)
pandas.core.frame.DataFrame
>>> type(res)
pandas.core.frame.DataFrame
```
To be able to use dask's shuffle operations with `dask_geopandas`, which uses a pandas subclass as the partition type, the subclass should be preserved in the partd roundtrip (or are there other ways that you can override / dispatch this operation in dask?).
I was wondering how other dask.dataframe subclasses handle this, but eg `dask_cudf` doesn't seem to support "disk"-based shuffling.
Contributor guide
Research direction
Start at the partd.PandasBlocks(partd.Dict()) entry point shown in the example and trace its append/get roundtrip. Done means a stored pandas DataFrame subclass, such as the GeoPandas example, is returned with its subclass preserved and the dask shuffle use case works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100