Geopandas vs pandas for points
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 394
- Forks
- 95
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 7
Description
Points now use lazy dataframes (Dask DataFrame). We talked about allowing having in-memory both as dataframes and lazy dataframes. https://github.com/scverse/spatialdata/issues/153
What about using GeoDataFrame and Dask GeoDataFrame instead? This will allow for:
- lazy loading
- unify points and circles (circles are in reality points in which we add a radius column via the schema) https://github.com/scverse/spatialdata/issues/46
- spatial index
- the user has to convert to
GeoDataFrameanyway to exploit thegeopandasfunctions, these functions are cumbersome (see below), but even worst, the user may be tempted or may expect to save points in theSpatialDataobject asGeoDataFrame.
Drawbacks:
- performance could not be good enough because it seems that
GeoDataFrameis creating one Python object per row, we should aim at 500M-1B points. - more complex type for the user
- 3D only partially supported by
geopandas, we need to understand the implication. Maybe it's fine to have the user only being able to do queries withgeopandasfor the 2D component of the data and use our APIs for 3D queries. In the end, if we use Dask dataframes we still need to implement these queries, so at worst we can extra queries APIs for free fromgeopandas.
Functions to convert back and forth points represented as Dask dataframes and geopandas dataframes (edit: an improved version of these functions is now available in the library, but their use is still cumbersome func 1, func 2).
from dask.dataframe.core import DataFrame as DaskDataFrame
from geopandas import GeoDataFrame
def points_dask_dataframe_to_geopandas(points: DaskDataFrame) -> GeoDataFrame:
# let's ignore the z component here
points_gdf = GeoDataFrame(geometry=geopandas.points_from_xy(points["x"], points["y"]))
for c in points.columns:
points_gdf[c] = points[c]
return points_gdf
def points_geopandas_to_dask_dataframe(gdf: GeoDataFrame) -> DaskDataFrame:
# convert the GeoDataFrame to a Dask DataFrame
ddf = dd.from_pandas(gdf[gdf.columns.drop("geometry")], npartitions=1)
ddf["x"] = gdf.geometry.x
ddf["y"] = gdf.geometry.y
# parse
ddf = PointsModel.parse(ddf, coordinates={"x": "x", "y": "y"})
return ddf
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the linked discussions in issues 153 and 46, then inspect the existing points_dask_dataframe_to_geopandas and points_geopandas_to_dask_dataframe functions documented in the issue. Compare the proposed GeoDataFrame approach against the stated performance, 3D support, lazy-loading, and circles requirements; done requires an agreed direction and a scoped implementation plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100