Using Big Data with Altair

I wanted to reach out, since we have been working on a similar project over at https://github.com/Quansight/jupyterlab-omnisci.

The goal there is to let users create Altair charts and have the heavy lifting transparently executed on a database.

To get a feel for it, you can [open the `notebooks/Ibis + Altair + Extraction.ipynb` notebook in Binder](https://mybinder.org/v2/gh/Quansight/jupyterlab-omnisci/master?urlpath=lab/). If you run the cells, the graphs should appear.

We are using [Ibis](https://github.com/ibis-project/ibis#ibis-python-data-analysis-framework-for-hadoop-and-sql-engines) to build up the SQL expression. We are building it to execute on an OmniSci database, but most of the work should translate to any other Ibis backend.

Currently, we update the Vega Lite spec to take out the transforms and [map them to Ibis](https://github.com/Quansight/jupyterlab-omnisci/blob/30c75ef7dc972d1aa9a493de500c41d4632f6950/jupyterlab_omnisci/altair.py#L321-L376). So we are implementing a very limited version of what you have here, targeting Ibis instead of Pandas, and using the extracted transforms in the VL spec.

However, our next goal is to support interactions, so that after a user interacts, a new query is computed and run. To do this, we are looking to switch from processing the Vega Lite spec to using the underlying Vega spec or graph. The idea being, we take the initial Altair chart, generate Vega Lite, convert to Vega, then pre-process the Vega spec to turn some of the transforms into a custom transform that will run the query using Ibis back on the kernel. We are tracking that here: https://github.com/Quansight/jupyterlab-omnisci/issues/54

On the Python side, that would involve somehow taking an existing Vega dataflow graph or Vega spec and understanding how those operations map to Ibis expressions. It seems that task shares a lot in common with what you have implemented here.

Like I said, although this work initially targets OmniSci, and their database is particular suited to computing these types of analytic queries, I hope that the general approach will be useful generally for using Altair in Python with other data sources on the kernel, like Pandas dataframes or other databases.

I would be happy to collaborate on any part of this that you would like or get your feedback on your general approach and understand if you have thoughts on how to support this kind of use case on top of Altair.

Also, thank you for helping to maintain this repo!

It's a treat to be able to use the UX in Altair to create large scale visualizations.