holoviz / holoviz/hvplot

Using Geopandas with hvPlot doesn't work for a large dataset.

Open
#1,341 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.4k
Forks
124
Avg merge
1d 18h
Merged PRs (30d)
1

Description

### ALL software version info
- MacOS - Sonoma 14.0
- python = 3.11.8
- notebook = 7.0.8
- dask = 2023.11.0
- datashader = 0.16.0
- geopandas = 0.14.2
- hvplot = 0.9.2
- holoviews = 1.18.3
- spatialpandas = 0.4.10
- pyarrow = 14.0.2
- pandas = 2.2.1
- bokeh = 3.4.0

### Description of expected behavior and the observed behavior

As part of the NumFOCUS SDG, I am modernizing the [nyc_building](https://examples.holoviz.org/gallery/nyc_buildings/nyc_buildings.html) example on the [examples website](https://examples.holoviz.org/gallery/index.html) to use the latest APIs. This example also involves using `geopandas` instead of `spatialpandas` to read the data stored as a parquet (`.parq`) file.

Switching from `spatialpandas` to `geopandas` for reading the file was not straightforward. The `geometry` column in the data file was not recognized by `geopandas`. To address this, I read the file using `spatialpandas`, converted it to a pandas DataFrame using `.compute()`, and then transformed it into a `geopandas` DataFrame. The `geometry` column was converted to a `shapely` object using a custom function. Finally, I saved this new DataFrame as a `.parq` file, which was then read directly using `geopandas`.

**Code:**
```python
import geopandas as gpd
import spatialpandas as spd
import spatialpandas.io

ddf = spd.io.read_parquet_dask('./data/nyc_buildings.parq').persist().compute()

gdf = gpd.GeoDataFrame(ddf)

def convert_to_shapely(spgeom):
try:
return spgeom.to_shapely()
except Exception as e:
print(f"Error converting geometry: {e}")
return None

gdf['geometry'] = gdf['geometry'].apply(convert_to_shapely)
gdf = gpd.GeoDataFrame(gdf, geometry='geometry')

gdf.to_parquet('new_nyc_buildings.parq')

gdf = gpd.read_parquet('new_nyc_buildings.parq')

gdf.head()
```

Despite these adjustments, plotting the new data file proved challenging. Using `hvPlot.polygons` to plot the entire dataset takes over 5 minutes to run with minimal code:

```python
gdf.hvplot.polygons(tiles='CartoLight', rasterize=True)
```

Testing with a small sample of the data (`gdf.head(1000)`) produced results in a reasonable time, suggesting that `hvPlot` may have difficulties handling the large dataset (over 1 million rows).

Additionally, plotting with extra parameters, such as adding color mapping to the `type` categories in the data results in a Traceback error.

### Complete, Minimal, Self-Contained Example Code that Reproduces the Issue

```python
gdf.head(1000).hvplot.polygons(tiles='CartoLight', rasterize=True, c='type', cmap=color_key)
```

### Stack traceback and/or browser JavaScript console output

```python
WARNING:param.dynamic_operation: Callable raised "ValueError('input must be numeric')".
Invoked as dynamic_operation(height=300, scale=1.0, width=700, x_range=None, y_range=None)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/IPython/core/formatters.py:974, in MimeBundleFormatter.__call__(self, obj, include, exclude)
971 method = get_real_method(obj, self.print_method)
973 if method is not None:
--> 974 return method(include=include, exclude=exclude)
975 return None
976 else:

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/dimension.py:1286, in Dimensioned._repr_mimebundle_(self, include, exclude)
1279 def _repr_mimebundle_(self, include=None, exclude=None):
1280 """
1281 Resolves the class hierarchy for the class rendering the
1282 object using any display hooks registered on Store.display
1283 hooks. The output of all registered display_hooks is then
1284 combined and returned.
1285 """
-> 1286 return Store.render(self)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/options.py:1428, in Store.render(cls, obj)
1426 data, metadata = {}, {}
1427 for hook in hooks:
-> 1428 ret = hook(obj)
1429 if ret is None:
1430 continue

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:287, in pprint_display(obj)
285 if not ip.display_formatter.formatters['text/plain'].pprint:
286 return None
--> 287 return display(obj, raw_output=True)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:261, in display(obj, raw_output, **kwargs)
259 elif isinstance(obj, (HoloMap, DynamicMap)):
260 with option_state(obj):
--> 261 output = map_display(obj)
262 elif isinstance(obj, Plot):
263 output = render(obj)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:149, in display_hook..wrapped(element)
147 try:
148 max_frames = OutputSettings.options['max_frames']
--> 149 mimebundle = fn(element, max_frames=max_frames)
150 if mimebundle is None:
151 return {}, {}

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:209, in map_display(vmap, max_frames)
206 max_frame_warning(max_frames)
207 return None
--> 209 return render(vmap)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:76, in render(obj, **kwargs)
73 if renderer.fig == 'pdf':
74 renderer = renderer.instance(fig='png')
---> 76 return renderer.components(obj, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/plotting/renderer.py:396, in Renderer.components(self, obj, fmt, comm, **kwargs)
393 embed = (not (dynamic or streams or self.widget_mode == 'live') or config.embed)
395 if embed or config.comms == 'default':
--> 396 return self._render_panel(plot, embed, comm)
397 return self._render_ipywidget(plot)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/plotting/renderer.py:403, in Renderer._render_panel(self, plot, embed, comm)
401 doc = Document()
402 with config.set(embed=embed):
--> 403 model = plot.layout._render_model(doc, comm)
404 if embed:
405 return render_model(model, comm)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/viewable.py:736, in Viewable._render_model(self, doc, comm)
734 if comm is None:
735 comm = state._comm_manager.get_server_comm()
--> 736 model = self.get_root(doc, comm)
738 if self._design and self._design.theme.bokeh_theme:
739 doc.theme = self._design.theme.bokeh_theme

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/layout/base.py:320, in Panel.get_root(self, doc, comm, preprocess)
316 def get_root(
317 self, doc: Optional[Document] = None, comm: Optional[Comm] = None,
318 preprocess: bool = True
319 ) -> Model:
--> 320 root = super().get_root(doc, comm, preprocess)
321 # ALERT: Find a better way to handle this
322 if hasattr(root, 'styles') and 'overflow-x' in root.styles:

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/viewable.py:667, in Renderable.get_root(self, doc, comm, preprocess)
665 wrapper = self._design._wrapper(self)
666 if wrapper is self:
--> 667 root = self._get_model(doc, comm=comm)
668 if preprocess:
669 self._preprocess(root)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/layout/base.py:186, in Panel._get_model(self, doc, root, parent, comm)
184 root = root or model
185 self._models[root.ref['id']] = (model, parent)
--> 186 objects, _ = self._get_objects(model, [], doc, root, comm)
187 props = self._get_properties(doc)
188 props[self._property_mapping['objects']] = objects

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/layout/base.py:168, in Panel._get_objects(self, model, old_objects, doc, root, comm)
166 else:
167 try:
--> 168 child = pane._get_model(doc, root, model, comm)
169 except RerenderError as e:
170 if e.layout is not None and e.layout is not self:

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/pane/holoviews.py:429, in HoloViews._get_model(self, doc, root, parent, comm)
427 plot = self.object
428 else:
--> 429 plot = self._render(doc, comm, root)
431 plot.pane = self
432 backend = plot.renderer.backend

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/pane/holoviews.py:525, in HoloViews._render(self, doc, comm, root)
522 if comm:
523 kwargs['comm'] = comm
--> 525 return renderer.get_plot(self.object, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/plotting/bokeh/renderer.py:68, in BokehRenderer.get_plot(self_or_cls, obj, doc, renderer, **kwargs)
61 @bothmethod
62 def get_plot(self_or_cls, obj, doc=None, renderer=None, **kwargs):
63 """
64 Given a HoloViews Viewable return a corresponding plot instance.
65 Allows supplying a document attach the plot to, useful when
66 combining the bokeh model with another plot.
67 """
---> 68 plot = super().get_plot(obj, doc, renderer, **kwargs)
69 if plot.document is None:
70 plot.document = Document() if self_or_cls.notebook_context else curdoc()

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/plotting/renderer.py:217, in Renderer.get_plot(self_or_cls, obj, doc, renderer, comm, **kwargs)
214 raise SkipRendering(msg.format(dims=dims))
216 # Initialize DynamicMaps with first data item
--> 217 initialize_dynamic(obj)
219 if not renderer:
220 renderer = self_or_cls

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/plotting/util.py:270, in initialize_dynamic(obj)
268 continue
269 if not len(dmap):
--> 270 dmap[dmap._initial_key()]

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:1217, in DynamicMap.__getitem__(self, key)
1215 # Not a cross product and nothing cached so compute element.
1216 if cache is not None: return cache
-> 1217 val = self._execute_callback(*tuple_key)
1218 if data_slice:
1219 val = self._dataslice(val, data_slice)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:984, in DynamicMap._execute_callback(self, *args)
981 kwargs['_memoization_hash_'] = hash_items
983 with dynamicmap_memoization(self.callback, self.streams):
--> 984 retval = self.callback(*args, **kwargs)
985 return self._style(retval)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:552, in Callable.__call__(self, *args, **kwargs)
550 return self.callable.rx.value
551 elif not args and not kwargs and not any(kwarg_hash):
--> 552 return self.callable()
553 inputs = [i for i in self.inputs if isinstance(i, DynamicMap)]
554 streams = []

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/util/__init__.py:1037, in Dynamic._dynamic_operation..dynamic_operation(*key, **kwargs)
1036 def dynamic_operation(*key, **kwargs):
-> 1037 key, obj = resolve(key, kwargs)
1038 return apply(obj, *key, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/util/__init__.py:1026, in Dynamic._dynamic_operation..resolve(key, kwargs)
1024 elif isinstance(map_obj, DynamicMap) and map_obj._posarg_keys and not key:
1025 key = tuple(kwargs[k] for k in map_obj._posarg_keys)
-> 1026 return key, map_obj[key]

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:1217, in DynamicMap.__getitem__(self, key)
1215 # Not a cross product and nothing cached so compute element.
1216 if cache is not None: return cache
-> 1217 val = self._execute_callback(*tuple_key)
1218 if data_slice:
1219 val = self._dataslice(val, data_slice)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:984, in DynamicMap._execute_callback(self, *args)
981 kwargs['_memoization_hash_'] = hash_items
983 with dynamicmap_memoization(self.callback, self.streams):
--> 984 retval = self.callback(*args, **kwargs)
985 return self._style(retval)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:552, in Callable.__call__(self, *args, **kwargs)
550 return self.callable.rx.value
551 elif not args and not kwargs and not any(kwarg_hash):
--> 552 return self.callable()
553 inputs = [i for i in self.inputs if isinstance(i, DynamicMap)]
554 streams = []

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/overlay.py:34, in Overlayable.__mul__..dynamic_mul(*args, **kwargs)
33 def dynamic_mul(*args, **kwargs):
---> 34 element = other[args]
35 return self * element

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:1217, in DynamicMap.__getitem__(self, key)
1215 # Not a cross product and nothing cached so compute element.
1216 if cache is not None: return cache
-> 1217 val = self._execute_callback(*tuple_key)
1218 if data_slice:
1219 val = self._dataslice(val, data_slice)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:984, in DynamicMap._execute_callback(self, *args)
981 kwargs['_memoization_hash_'] = hash_items
983 with dynamicmap_memoization(self.callback, self.streams):
--> 984 retval = self.callback(*args, **kwargs)
985 return self._style(retval)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:582, in Callable.__call__(self, *args, **kwargs)
579 args, kwargs = (), dict(pos_kwargs, **kwargs)
581 try:
--> 582 ret = self.callable(*args, **kwargs)
583 except KeyError:
584 # KeyError is caught separately because it is used to signal
585 # invalid keys on DynamicMap and should not warn
586 raise

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/util/__init__.py:1038, in Dynamic._dynamic_operation..dynamic_operation(*key, **kwargs)
1036 def dynamic_operation(*key, **kwargs):
1037 key, obj = resolve(key, kwargs)
-> 1038 return apply(obj, *key, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/util/__init__.py:1030, in Dynamic._dynamic_operation..apply(element, *key, **kwargs)
1028 def apply(element, *key, **kwargs):
1029 kwargs = dict(util.resolve_dependent_kwargs(self.p.kwargs), **kwargs)
-> 1030 processed = self._process(element, key, kwargs)
1031 if (self.p.link_dataset and isinstance(element, Dataset) and
1032 isinstance(processed, Dataset) and processed._dataset is None):
1033 processed._dataset = element.dataset

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/util/__init__.py:1012, in Dynamic._process(self, element, key, kwargs)
1010 elif isinstance(self.p.operation, Operation):
1011 kwargs = {k: v for k, v in kwargs.items() if k in self.p.operation.param}
-> 1012 return self.p.operation.process_element(element, key, **kwargs)
1013 else:
1014 return self.p.operation(element, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/operation.py:194, in Operation.process_element(self, element, key, **params)
191 else:
192 self.p = param.ParamOverrides(self, params,
193 allow_extra_keywords=self._allow_extra_keywords)
--> 194 return self._apply(element, key)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/operation.py:141, in Operation._apply(self, element, key)
139 if not in_method:
140 element._in_method = True
--> 141 ret = self._process(element, key)
142 if hasattr(element, '_in_method') and not in_method:
143 element._in_method = in_method

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/operation/datashader.py:1506, in rasterize._process(self, element, key)
1503 op = transform.instance(**{k:v for k,v in extended_kws.items()
1504 if k in transform.param})
1505 op._precomputed = self._precomputed
-> 1506 element = element.map(op, predicate)
1507 self._precomputed = op._precomputed
1509 unused_params = list(all_supplied_kws - all_allowed_kws)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/data/__init__.py:196, in PipelineMeta.pipelined..pipelined_fn(*args, **kwargs)
193 inst._in_method = True
195 try:
--> 196 result = method_fn(*args, **kwargs)
197 if PipelineMeta.disable:
198 return result

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/data/__init__.py:1213, in Dataset.map(self, *args, **kwargs)
1211 @wraps(LabelledData.map)
1212 def map(self, *args, **kwargs):
-> 1213 return super().map(*args, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/dimension.py:697, in LabelledData.map(self, map_fn, specs, clone)
695 return deep_mapped
696 else:
--> 697 return map_fn(self) if applies else self

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/operation.py:214, in Operation.__call__(self, element, **kwargs)
210 return element.clone([(k, self._apply(el, key=k))
211 for k, el in element.items()])
212 elif ((self._per_element and isinstance(element, Element)) or
213 (not self._per_element and isinstance(element, ViewableElement))):
--> 214 return self._apply(element)
215 elif 'streams' not in kwargs:
216 kwargs['streams'] = self.p.streams

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/operation.py:141, in Operation._apply(self, element, key)
139 if not in_method:
140 element._in_method = True
--> 141 ret = self._process(element, key)
142 if hasattr(element, '_in_method') and not in_method:
143 element._in_method = in_method

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/operation/datashader.py:1391, in geometry_rasterize._process(self, element, key)
1389 agg_kwargs = dict(geometry=col, agg=agg_fn)
1390 if isinstance(element, Polygons):
-> 1391 agg = cvs.polygons(data, **agg_kwargs)
1392 elif isinstance(element, Path):
1393 if self.p.line_width and ds_version >= Version('0.14.0'):

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/datashader/core.py:782, in Canvas.polygons(self, source, geometry, agg)
780 if agg is None:
781 agg = any_rdn()
--> 782 return bypixel(source, self, glyph, agg)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/datashader/core.py:1329, in bypixel(source, canvas, glyph, agg, antialias)
1327 schema = dshape.measure
1328 glyph.validate(schema)
-> 1329 agg.validate(schema)
1330 canvas.validate()
1332 # All-NaN objects (e.g. chunks of arrays with no data) are valid in Datashader

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/datashader/reductions.py:355, in Reduction.validate(self, in_dshape)
353 raise ValueError("specified column not found")
354 if not isnumeric(in_dshape.measure[self.column]):
--> 355 raise ValueError("input must be numeric")

ValueError: input must be numeric
:DynamicMap []
```

### Screenshots or screencasts of the bug in action

Screenshot 2024-05-15 at 5 00 24 PM

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.