greta-dev / greta-dev/greta

Discussion) A generalization of `greta`'s DAG construction for matrices/arrays

Open
#247 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
607
Forks
67
Avg merge
3d 8h
Merged PRs (30d)
1

Description

Hi @goldingn, I've been reading through some of `greta`'s source code, and I'm pretty impressed with the R6 setup for nodes of the DAG. I've been thinking a bit about how that could generalize to be used outside of `greta`, and wanted to get your thoughts.

Imagine a world where instead of executing `z <- x + y` on two R matrices immediately, we instead capture the fact that you are doing addition on 2 objects and return something that has the correct shape of the output, but with `?` in each cell rather than the result.`greta` essentially does this already. Then, the user can run `calculate()` on `z` to actually get the result. This can continue, meaning that we can do `z + x` to get another object, without ever calculating `z`. You already know all this.

The best part, in my opinion, is that since you have captured just the operations, it is very much like `dplyr` in that the backend can be _anything_. It can be base R, tensorflow, or xtensor (a cpp library i've been working with that allows for lazy and broadcasted array semantics). Or even a database where the table is purely numeric.

So essentially what I propose is extracting out the node scaffolding from `greta`, making them a bit more generic (so `tf_operation` would just be `operation` which could store "add" and then you specify that you are using tensorflow later, or xtensor or whatever), and then `greta` is just one extension of that system.

What do you think? It's obviously a huge project, and nothing that I'd be requesting you do alone, but I think it could provide for some really powerful abstraction.

PS) I have a little work on an integration of xtensor into R in the `rray` package [here](https://github.com/DavisVaughan/rray). It doesn't take advantage of the fact that the operations are lazy though, which this abstraction could.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.