Option to error when "Assigning to n row subset of m rows" with n > m

Open
#2,022 2 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
r
Domain
data

Research direction

Reproduce the join assignment with b[a, on=.(id), x := i.x, verbose = TRUE] and compare it with the opposite join workaround shown in the issue. Investigate the join-assignment entry point that emits the verbose row-subset message. Done requires an agreed behavior for detecting non-unique matches and reporting them, since the issue leaves error, warning, and diagnostic-object approaches open.

Written by the indexing model from the issue text.

Description

joins

In a join, x[i, v := i.v], if multiple rows of i match to a single row of x, the assignment takes the last one (?). It would be nice to get an error or maybe a warning when this behavior is triggered.

library(data.table)
a <- data.table(id = c(1L, 1L, 2L, 3L, NA_integer_), x = 11:15)
b <- data.table(id = 1:2, y = -(1:2))
b[a, on=.(id), x := i.x, verbose = TRUE]
# Calculated ad hoc index in 0 secs
# Starting bmerge ...done in 0 secs
# Detected that j uses these columns: x,i.x 
# Assigning to 3 row subset of 2 rows

I'm not sure if the condition in the title (n > m) is necessary and sufficient for this behavior, though.

My workaround for now would involve looking at the opposite join:

a[b, on=.(id), .N, by=.EACHI][, range(N)]
# [1] 1 2

That seems pretty cumbersome. Maybe there's some way for me to capture and grep the verbose output (but then again, maybe not).


Just an idea: A more general approach could involve returning an object containing diagnostics from the join and assignment. Of course, the object cannot be the return value of [.data.table, but maybe it could be dropped in some locked-binding global, .datatable.diagnostic similar to .Last.value. Alternately, maybe that sort of object would fit well into @jangorecki 's dtq package.

I'm thinking along these lines as I write tutorial materials to convert Stata users to R. In Stata, all joins cat a nice-ish table to the console.

SO post from a Stata user interested in uniqueness of matching of each row of i in x etc: https://stackoverflow.com/questions/49541330/r-data-table-merge-vs-stata-merge


Update: Re the verbose message text, the n is recorded thanks to https://github.com/Rdatatable/data.table/pull/3460 and the m is just the number of rows in the table (which I guess I didn't realize at the time I posted this, thinking it was instead m = uniqueN(irows, nar.m = TRUE)... which unfortunately is not computed, and there is no way to detect whether the update join was 1:1, etc per the SO link above).

So anyway, I'll leave this open since it seems to highlight a point of difficulty (judging by emoji-votes) even if my suggestion does not fix it.

Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.