Discussion) A generalization of `greta`'s DAG construction for matrices/arrays
- Dominant language
- C++
- Stars
- 607
- Forks
- 67
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 1
Description
Hi @goldingn, I've been reading through some of `greta`'s source code, and I'm pretty impressed with the R6 setup for nodes of the DAG. I've been thinking a bit about how that could generalize to be used outside of `greta`, and wanted to get your thoughts.
Imagine a world where instead of executing `z <- x + y` on two R matrices immediately, we instead capture the fact that you are doing addition on 2 objects and return something that has the correct shape of the output, but with `?` in each cell rather than the result.`greta` essentially does this already. Then, the user can run `calculate()` on `z` to actually get the result. This can continue, meaning that we can do `z + x` to get another object, without ever calculating `z`. You already know all this.
The best part, in my opinion, is that since you have captured just the operations, it is very much like `dplyr` in that the backend can be _anything_. It can be base R, tensorflow, or xtensor (a cpp library i've been working with that allows for lazy and broadcasted array semantics). Or even a database where the table is purely numeric.
So essentially what I propose is extracting out the node scaffolding from `greta`, making them a bit more generic (so `tf_operation` would just be `operation` which could store "add" and then you specify that you are using tensorflow later, or xtensor or whatever), and then `greta` is just one extension of that system.
What do you think? It's obviously a huge project, and nothing that I'd be requesting you do alone, but I think it could provide for some really powerful abstraction.
PS) I have a little work on an integration of xtensor into R in the `rray` package [here](https://github.com/DavisVaughan/rray). It doesn't take advantage of the fact that the operations are lazy though, which this abstraction could.
Contributor guide
Assessment
This issue has not been assessed yet.