nafill new type: approx
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start with the existing nafill entry point and its current types, then review how the examples use approx() and zoo::na.approx() for comparison. Define the expected behavior for the proposed approx type, including the missing-value runs shown in the issue, and validate the result against the supplied examples and performance benchmark.
Written by the indexing model from the issue text.
Description
This is a FR. Also I apologize ahead of time if this is not an appropriate post for here.
From this StackOverflow post:
library(data.table)
dt <- data.table(dist = c(31091.33, NA, 31100.00, 31103.27, NA, NA, NA, NA, 31124.98))
dt[, time := .I]
## dist time
##1: 31091.33 1
##2: NA 2
##3: 31100.00 3
##4: 31103.27 4
##5: NA 5
##6: NA 6
##7: NA 7
##8: NA 8
##9: 31124.98 9
OP wants to do a linear interpolation to fill in the NA values. The most simple solution is to use base approx() or zoo:na.approx() per user dww:
copy(dt)[, dist := zoo::na.approx(dist)][]
copy(dt)[, dist := approx(.I, dist, .I)$y][]
# dist time
#1: 31091.33 1
#2: 31095.67 2
#3: 31100.00 3
#4: 31103.27 4
#5: 31107.61 5
#6: 31111.95 6
#7: 31116.30 7
#8: 31120.64 8
#9: 31124.98 9
The current nafill types are: c("const", "locf", "nocb") - an "approx" type would be helpful. I wrote an Rcpp implementation to show that there is room for improvement as it's ~10 times faster than the base method which is already a C based method:
#data per @chinsoon12
nr <- 1e7
nNA <- nr/2
DT <- data.table(time=1:nr, dist=replace(rnorm(nr), sample(1:nr, nNA), NA_real_))
bench::mark(
rcpp_appr = rcpp_approx(DT[['dist']]),
base_approx = DT[, approx(.I, dist, .I)$y],
zoo_approx = zoo::na.approx(DT[['dist']]),
time_unit = 's'
)
# A tibble: 3 x 13 (in seconds)
expression min median `itr/sec` mem_alloc
<bch:expr> <dbl> <dbl> <dbl> <bch:byt>
1 rcpp_appr 0.126 0.132 7.08 76MB
2 base_approx 1.34 1.34 0.745 617MB
3 zoo_approx 2.22 2.22 0.450 1630MB ##manually changed from GB to MB
# this is the rcpp code
Rcpp::sourceCpp(code = '
#include <Rcpp.h>
using namespace Rcpp;
// [[Rcpp::export]]
NumericVector rcpp_approx(NumericVector y) {
double start = 0, slope = 0;
int count = 0;
NumericVector y1 = clone(y); //added to not update-by-reference
for(int i = 0; i < y1.size(); ++i){
if (NumericVector::is_na(y1[i])){
count++;
} else {
if (count != 0) {
start = y1[i - (count+1)];
slope = (y1[i] - start) / (count + 1);
for (int j = 0; j < count; j++){
y1[i-(count-j)] = start + slope * (j + 1);
}
count = 0;
}
}
}
return(y1);
}
')
Thank you for your consideration.
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·