DaskR prototype with rpy2 and reticulate
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
@quasiben has been playing with Dask and R with [reticulate](https://github.com/rstudio/reticulate) and [rpy2](https://rpy2.readthedocs.io/) with the objective of providing Dask's concurrent.futures API to R (from which they could presumably build other systems). This is somewhat tricky because you have both a Python and R session living side-by-side and need to move things between them from time to time (hopefully infrequently) using reticulate or rpy2. There are a variety of ways to do this, I thought I'd lay out the way that makes the most sense to me.
- From R we need ways to get proxy objects in Python that point to our local objects in R without doing any clever/lossy conversion (like R dataframes to Pandas dataframes). Below I use rpy2 as this proxy object, but anything would do.
- From within Python we need ways to serialize and deserialize these proxy objects, presumably this involves calling R's `serialize` function on the R side and then bringing those bytes back to Python, and then doing the reverse with deserialize.
On the Dask side we need to implement the following interface for `rpy2` objects: https://distributed.readthedocs.io/en/latest/serialization.html#dask-serialization-family
```python
@dask_serialize.register(rpy.whatever_an_object_type_is)
def serialize_rpy2(obj):
r_bytes = rpy2.call("serialize", obj) # call R `serialize` function on pointed-to-object
py_bytes = rpy2.get_from_R(r_bytes) # bring bytes back home to python
header = {}
frames = [py_bytes]
return header, frames
@dask_deserialize.register(rpy2.whatever_an_object_type_is)
def deserialize_rpy2(header, frames):
[py_bytes] = frames
r_bytes = rpy2.send_to_R(py_bytes)
obj = rpy2.call("deserialize", r_bytes)
return obj
```
- Also on the Python side we need to implement the `sizeof` protocol` to quickly estimate how large an object is in bytes
```python
from dask.sizeof import sizeof
@sizeof.register(rpy2.whatever_type_objects_are)
def sizeof(x) -> int:
return number_of_bytes
```
- We'll need to develop a basic API in R. If we're starting with the concurrent.futures interface (which seems simplest) then this probably means `submit`, `gather`, `as_completed`, `wait`, and `map` (is anything else critical?). We should check in with people familiar with R though to learn if there is a more native API that R users would find more familiar.
Contributor guide
Assessment
This issue has not been assessed yet.