dask / dask/distributed

DaskR prototype with rpy2 and reticulate

Open
#2,254 4 comments 0 reactions 0 assignees View on GitHub
discussion enhancement
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

@quasiben has been playing with Dask and R with [reticulate](https://github.com/rstudio/reticulate) and [rpy2](https://rpy2.readthedocs.io/) with the objective of providing Dask's concurrent.futures API to R (from which they could presumably build other systems). This is somewhat tricky because you have both a Python and R session living side-by-side and need to move things between them from time to time (hopefully infrequently) using reticulate or rpy2. There are a variety of ways to do this, I thought I'd lay out the way that makes the most sense to me.

- From R we need ways to get proxy objects in Python that point to our local objects in R without doing any clever/lossy conversion (like R dataframes to Pandas dataframes). Below I use rpy2 as this proxy object, but anything would do.
- From within Python we need ways to serialize and deserialize these proxy objects, presumably this involves calling R's `serialize` function on the R side and then bringing those bytes back to Python, and then doing the reverse with deserialize.

On the Dask side we need to implement the following interface for `rpy2` objects: https://distributed.readthedocs.io/en/latest/serialization.html#dask-serialization-family

```python
@dask_serialize.register(rpy.whatever_an_object_type_is)
def serialize_rpy2(obj):
r_bytes = rpy2.call("serialize", obj) # call R `serialize` function on pointed-to-object
py_bytes = rpy2.get_from_R(r_bytes) # bring bytes back home to python
header = {}
frames = [py_bytes]
return header, frames

@dask_deserialize.register(rpy2.whatever_an_object_type_is)
def deserialize_rpy2(header, frames):
[py_bytes] = frames
r_bytes = rpy2.send_to_R(py_bytes)
obj = rpy2.call("deserialize", r_bytes)
return obj
```

- Also on the Python side we need to implement the `sizeof` protocol` to quickly estimate how large an object is in bytes

```python
from dask.sizeof import sizeof
@sizeof.register(rpy2.whatever_type_objects_are)
def sizeof(x) -> int:
return number_of_bytes
```
- We'll need to develop a basic API in R. If we're starting with the concurrent.futures interface (which seems simplest) then this probably means `submit`, `gather`, `as_completed`, `wait`, and `map` (is anything else critical?). We should check in with people familiar with R though to learn if there is a more native API that R users would find more familiar.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.