holoviz / holoviz/datashader

Improving Datashader's API

Open
#490 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.6k
Forks
376
Avg merge
4h 32m
Merged PRs (30d)
1

Description

Datashader's API was initially designed mainly to help exercise the underlying computations, not to make things straightforward for the user. As the library has matured, the main thing keeping the version number below 1.0 has been the desire to make a cleaner API for end users before settling on it. This issue outlines a proposed new API that would be used to help datashader reach 1.0.

## Current API

The current API for making an image from a dataframe is a bit muddled:

```
cvs = ds.Canvas(plot_width=200, plot_height=200, x_range=(-8,8), y_range=(-8,8))
agg = cvs.points(df,'x','y',agg=ds.reductions.count())
img = tf.shade(agg)
```

(There is also another way to make an agg, using ``agg = ds.bypixel(df, cvs, ds.glyphs.Point('x','y'), ds.reductions.count())``, but this syntax only works for lines and points and is even more confusing for the end user, so we'll ignore it and hopefully delete mentions of it until that too can be cleaned up. Also ignoring Pipeline, which is not widely used and can probably be deleted.)

The above formulation is at odds with the pipeline diagram laid out in the pipeline.ipynb documentation:

![pipeline diagram](https://raw.githubusercontent.com/bokeh/datashader/master/docs/images/pipeline2.png)

The advertised pipeline starts with a "Scene" object that is somewhere in between the Canvas and aggregate objects here -- it includes everything in Canvas, plus the specification of a specific glyph that's currently not known until the ``points()`` (or ``line()`` or ``raster()``) call. Introducing the glyph and the columns so late is odd, because those are the key bits of metadata that determines what any of this plot will be about. For a given pair of columns, the glyph type is arguably an implicit property of the dataframe itself, i.e. a declaration of what that data represents. Introducing such key metadata only at the moment of aggregation seems backwards; we'll never normally want to change that for an aggregation call of a particular set of columns . Conversely, the width and height are properties of the resulting aggregation, and so it's odd for them not to be specifiable in the call that actually generates the aggregation. So it seems to be divided up between stages in a confusing and not very helpful way that makes it difficult to describe just what Canvas is or does. For these reasons, in practice, people end up building a Canvas and making an aggregation as a single step, so having Canvas as some separate object is not currently achieving anything obvious.

## Proposed User API 1.0:

Combine the Canvas object and the canvas glyph methods into three new classes ``ds.Points``, ``ds.Lines``, and ``ds.Raster``, each inheriting from a new superclass ``ds.Scene``. Each of the ``Scene`` classes will contain all data and specifications needed to create an aggregate, but these values can be overridden during an aggregate() call if desired:

```
scene = ds.Points(df,'x','y',x_range=(-8,8), y_range=(-8,8),agg=ds.reductions.count()))
agg = scene.aggregate()
img = tf.shade(agg)
````

Here the user is forced to choose a glyph type at the same time as supplying the dataframe and column names, which makes sense semantically to me and is certainly the set of information that *must* always be provided from the user. Any other information can be specified or overridden by the user at any level they prefer:

- The Scene class defines common parameters that can be set as class attributes at this superclass level, applying to all Scene types
- Specific Scene subclasses define additional parameters or set defaults specific to that type of Scene that can be set or changed as class attributes at that level
- A specific Scene class can be instantiated with specific parameter values
- An aggregate() call can override any of these values
- shade() takes its own options applying only to that stage, with similar Parameterized support

In this way, a user can specify what is known from the start (that the data represents Points or Lines or a Raster), and can then change only whatever is needed to change in a specific
aggregation step (typically ranges, resolutions, and aggregator). See partial implementation below.

Here I've used the method name ``aggregate()`` for concreteness, but we could presumably use ``__call__``, since aggregating is the one obvious operation to be done on a Scene.

## Proposed User API extension: Rich display for Scenes

Once the above API has been implemented, the Scene object will now contain *all* of the information necessary to render a default image, as you can see by the fact that the ``aggregate()`` and ``shade`` calls don't require any arguments. Thus we can consider making the Scene object visualize itself in a Jupyter notebook by default, just as the Image objects returned by tf.shade do, reducing the minimal invocation for a datashader image to just:

```
ds.Points(df,'x','y')
```

(which would render using all the default options to a PNG visible with Jupyter's rich display support). This seems convenient, but it does seem difficult to know where to stop, because we could then make this display be configurable by making tf.shade be a Parameterized object and instantiating it in the Scene class, allowing the user to change its parameters if desired. And then people would want to define an optional transformation step on the aggregate, dynamic spreading on the final result, and so on, ending up with another version of Pipeline.

Perhaps it would be safer to provide somewhat similar levels of convenience by making ``shade`` be an operation like those in holoviews.operation.datashader:

```
shade(ds.Points(df,'x','y'))
```

In this way shade can accept its own options, while still conveniently displaying in a notebook.

For this to work, ``shade`` would have to check to see if it's been given a Scene rather than an xarray, and would call it to do the aggregation first if need be. Without that extra code, it's a bit uglier, but not too bad assuming we use ``__call__`` syntax:

```
shade(ds.Points(df,'x','y')())
```

## Current Implementation

The various glyph types (points, lines, and rasters) are defined as objects in datashader.glyph, but they are actually typically accessed as methods on a Canvas object that doesn't do much other than access them:

```
class Canvas(object):
def __init__(self, plot_width=600, plot_height=600, x_range=None, y_range=None, x_axis_type='linear', y_axis_type='linear'):
def points(self, source, x, y, agg=None):
def line(self, source, x, y, agg=None):
def raster(self, source, layer=None, upsample_method='linear', downsample_method='mean', nan_value=None):
```

## Proposed Implementation

```
class Scene(Parameterized):
width = param.Integer(default=600, doc="Width of aggregate array to create")
height = param.Integer(default=600, doc="Height of aggregate array to create")
x_range = param.NumericTuple(default=None)
y_range = param.NumericTuple(default=None)
x_axis_type = param.ObjectSelector(default='linear',objects=['linear','log'])
y_axis_type = param.ObjectSelector(default='linear',objects=['linear','log'])
agg = param.ClassSelector(class_=Reduction, default=None)

def __init__(self,source,**params):
self.source = source

def aggregate(self,**params):
p=paramOverrides(params)
return bypixel(self.source, self, self.glyph(x, y), p.agg)

class Points(Scene):
agg = param.ClassSelector(count())
glyph = glyph.Point

class Lines(Scene):
agg = param.ClassSelector(any())
glyph = glyph.Line

class Raster(Scene):
agg = param.ClassSelector(mean())
interpolator = param.ClassSelector(class_=Upsample, default=linear())
nan_value = param.Number(default=None)
layer = param.Integer(default=None)

def aggregate(self):
....
```

## Migration Path

The new Scene objects would go into datashader/scene.py, and would be imported into the top level. scene.py would currently import the required implementation from core.py.
These objects shoudn't have any effect on the existing Canvas object, which can be retained while we remove all instances of it (and of Pipeline) from examples. At that point
Canvas can be deprecated and eventually removed.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.