Proposal: new vector representation `RepType::Closure`
- Dominant language
- Rust
- Stars
- 145
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
**NOTE: This proposal is superseded by the first comment in this Issue.**
Sometimes, there are cases where we want to apply a function lazily to a vector.
One example is in the internal code-base, where we would like to cast an `(Obj, Obj)` tuple to their minimally numeric type.
Currently, this requires us to allocate a new vector.
My suggestion here is to allow for lazy computation on vectors by adding a new variant to `RepType`, the **`Map`** variant:
```rust
enum RepType {
Subset(CowObj>, Subsets),
Map(RepType, Box T>)
}
```
If we wanted to cast a tupe `(Obj::Vector(Vector::Integer(_)), Obj::Vector(Vector::Double(_))` to their minimally numeric type, we could lazily transform the first element of the tuple to a `Obj::Vector(Vector::Double(_))`, by converting the underlying vector `v` (a `RepType::Subset`) to a `RepType::Map(v, |i| i.map(i as f64))`.
Of course, this `RepType::Map` would also be generally useful to avoid any intermediate vector allocations, such as in the example below, where we calculate the variance of a random vector. In R, this would first allocate the vector `tmp1 = x - mean(x)`, then allocate the vector `tmp2 = tmp1^2`, only to then calculate `mean(tmp2)`. If we were to add the `RepType::Map`, we could avoid both allocations.
```r
x = rnorm(100)
mean((x - mean(x))^2) / length(x)
```
Further, this would also make the need for "fused" functions such as `anyNA` unnecessary. In R, calling `any(is.na(x))` is slow, because `is.na(x)` allocates a new logical vector. But if `is.na(x)` produces a `RepType::Map`, this should not be such a big problem.
---
*Where this Suggestion falls short*
One disadvantage of the `RepType::Map` is that it does not allow to combine different vectors.
Let's say, we want to calculate the covariance between two random vectors:
```r
x = rnorm(100)
y = x * 0.2 + rnorm(100) *0.3
mean((x - mean(x))(y - mean(x))) / 100
```
Here, we first calculate `tmp1 = x - mean(x)` and `tmp2 = y - mean(x)` which can be represented both as `RepType::Map`s.
But if we then multiply them, we need to allocate a vector `tmp3`, just so we then calculate `sum(tmp3) / 100`.
This means that we might want to add another variant (e.g. calling it **`RepType::Merge`**) to represent such operations:
```rust
enum RepType {
..,
Merge(RepType, RepType, Box T>)
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Read the first comment, which supersedes the proposal, before investigating the requested RepType changes. Use the current discussion to determine the accepted scope for lazy vector operations and what outcome, if any, remains actionable; the issue should be updated or closed once the superseding direction is reflected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100