i column not found in rolling join with by
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 38/100
Research direction
Start by running the reproducible DT[i,j,by,roll] examples in the issue, especially regular[irregular, ...]. Compare the failing mean(Y) case with the working .N grouping and the alternative non-equi join. Done means the rolling join can aggregate Y with by=grid without materializing joined, producing the expected group counts and means.
Written by the indexing model from the issue text.
Description
hi all! I'm not an expert in rolling joins, but I thought it should be possible to use arguments by and roll at the same time to do rolling join and summarization efficiently in one step. But I observed a "not found" error when I tried that.
Here is a complete example:
library(data.table)
X.vec = c(0.05, 0.15, 0.75, 0.95)
Y.vec = 10*X.vec
(irregular = data.table(X=X.vec, Y=Y.vec, key="X"))
grid.space = 0.2
offset = grid.space/2
grid = seq(offset, 1-offset, by=grid.space)
(regular <- data.table(grid, X=grid, min=grid-offset, max=grid+offset, key="X"))
(joined = regular[irregular, roll="nearest"])
joined[, .(N=.N, mean.Y=mean(Y)), by=grid]
irregular[regular, .(grid, N=.N, mean.Y=mean(Y)), on=.(X<max, X>min), by=.EACHI, nomatch=0L]
regular[irregular, .(N=.N), roll="nearest", by=grid]
regular[irregular, .(N=.N, mean.Y=mean(Y)), roll="nearest", by=grid]
The goal is a rolling join between regular and irregular, with matching each irregular$X to the closest regular$grid value, then computing the mean of irregular$Y for each value of regular$grid.
> (irregular = data.table(X=X.vec, Y=Y.vec, key="X"))
X Y
1: 0.05 0.5
2: 0.15 1.5
3: 0.75 7.5
4: 0.95 9.5
> (regular <- data.table(grid, X=grid, min=grid-offset, max=grid+offset, key="X"))
grid X min max
1: 0.1 0.1 0.0 0.2
2: 0.3 0.3 0.2 0.4
3: 0.5 0.5 0.4 0.6
4: 0.7 0.7 0.6 0.8
5: 0.9 0.9 0.8 1.0
One way to do that is by first materializing the joined table, then summarizing it:
> (joined = regular[irregular, roll="nearest"])
grid X min max Y
1: 0.1 0.05 0.0 0.2 0.5
2: 0.1 0.15 0.0 0.2 1.5
3: 0.7 0.75 0.6 0.8 7.5
4: 0.9 0.95 0.8 1.0 9.5
> joined[, .(N=.N, mean.Y=mean(Y)), by=grid]
grid N mean.Y
1: 0.1 2 1.0
2: 0.7 1 7.5
3: 0.9 1 9.5
That works fine, but I thought there would be some more efficient way to do that (which could avoid allocating memory for the intermediate joined table). So I tried adding by to the same call as roll but I got an error:
> regular[irregular, .(N=.N, mean.Y=mean(Y)), roll="nearest", by=grid]
Error in `[.data.table`(regular, irregular, .(N = .N, mean.Y = mean(Y)), :
object 'Y' not found
Is this a bug? In other words, when doing DT[i,j,by,roll], shouldn't it be possible to use columns of i in j?
One thing that suggests that it should be possible is the code below, which shows that you can count the number of items in each by group:
> regular[irregular, .(N=.N), roll="nearest", by=grid]
grid N
1: 0.1 2
2: 0.7 1
3: 0.9 1
Finally, another way of computing the result that I want, in one step (without the intermediate table) would be:
> irregular[regular, .(grid, N=.N, mean.Y=mean(Y)), on=.(X<max, X>min), by=.EACHI, nomatch=0L]
X X grid N mean.Y
1: 0.2 0.0 0.1 2 1.0
2: 0.8 0.6 0.7 1 7.5
3: 1.0 0.8 0.9 1 9.5
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·