dgkf / dgkf/R

Representation of missing data

Open
#106 1 comment 0 reactions 0 assignees View on GitHub
theme-internals type-design type-enhancement
Dominant language
Rust
Stars
145
Forks
5
PR merge metrics
No merged PRs in 30d

Description

First of all, I hope it is okay that I spam all these issues ... 😄

I was wondering whether the representation of missing values might be reconsidered.
Currently, each element of a vector can be NA or not NA (https://github.com/dgkf/R/blob/2ef9780a2a7a155a22bd42cbad73d59a3b084324/src/object/vector/core.rs#L14)
However, most vectors will not contain any NAs at all.
For some operations, the loss in performance of using the current representation might be significant.
For example (I am by no means an expert in this topic, but I read [a great blog post](https://viralinstruction.com/posts/hardware/)), using SIMD registers requires branch-free operations. If we ever want to use SIMD instructions, these cannot be applied to the current implementation of the atomic vectors I believe.

An approach that at least superficially seems appealing to me is to not only specify on the element-level whether an element is NA or not, but to also specify for the whole vector whether it contains missing data or not.
In julia for example, missing values are represented using the `Missing` type, and mixing numeric values and missing data is achieved via a Union-Type:

```julia
julia> [1, 2, missing]
3-element Vector{Union{Missing, Int64}}:
```

One major difference is now that missing values cannot be assigned to vectors that are not of the union type.

```julia
julia> x = [1, 2, 3]
3-element Vector{Int64}:
1
2
3

julia> x[1] = missing
ERROR: MethodError: Cannot `convert` an object of type Missing to an object of type Int64
```

Concatenation of different types is possible however:

```julia

julia> [[1, 2]; [2, missing]]
4-element Vector{Union{Missing, Int64}}:
1
2
2
missing
```

Another difference between the missing representation in `R` and `julia` is that the latter has only one `Missing` type, whereas `R` has `NA_integer_` etc.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.