Representation of missing data
- Dominant language
- Rust
- Stars
- 145
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
First of all, I hope it is okay that I spam all these issues ... 😄
I was wondering whether the representation of missing values might be reconsidered.
Currently, each element of a vector can be NA or not NA (https://github.com/dgkf/R/blob/2ef9780a2a7a155a22bd42cbad73d59a3b084324/src/object/vector/core.rs#L14)
However, most vectors will not contain any NAs at all.
For some operations, the loss in performance of using the current representation might be significant.
For example (I am by no means an expert in this topic, but I read [a great blog post](https://viralinstruction.com/posts/hardware/)), using SIMD registers requires branch-free operations. If we ever want to use SIMD instructions, these cannot be applied to the current implementation of the atomic vectors I believe.
An approach that at least superficially seems appealing to me is to not only specify on the element-level whether an element is NA or not, but to also specify for the whole vector whether it contains missing data or not.
In julia for example, missing values are represented using the `Missing` type, and mixing numeric values and missing data is achieved via a Union-Type:
```julia
julia> [1, 2, missing]
3-element Vector{Union{Missing, Int64}}:
```
One major difference is now that missing values cannot be assigned to vectors that are not of the union type.
```julia
julia> x = [1, 2, 3]
3-element Vector{Int64}:
1
2
3
julia> x[1] = missing
ERROR: MethodError: Cannot `convert` an object of type Missing to an object of type Int64
```
Concatenation of different types is possible however:
```julia
julia> [[1, 2]; [2, missing]]
4-element Vector{Union{Missing, Int64}}:
1
2
2
missing
```
Another difference between the missing representation in `R` and `julia` is that the latter has only one `Missing` type, whereas `R` has `NA_integer_` etc.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.