Bug interacting with `data.table` embedded in list created by another package

Open
#1,310 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
30/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
r
Domain
data

Research direction

Start by running the reported maptools/sp example with setDT(us.states@data), then trace the [.data.table, setDT, and := behavior for a data.table stored in us.states@data. Done means the reproduction's new column is added correctly without the reported invalid .internal.selfref warning, with the behavior checked against the supplied example.

Written by the indexing model from the issue text.

Description

bug non-atomic column

As mentioned and ellaborated in this SO post, I've run into what appears to be a bug trying to get maptools/sp to cooperate with data.table.

Copy-pasting the example from the post (data here):

library(maptools)

us.states<-readShapePoly("cb_2014_us_state_5m.shp")

setDT(us.states@data) #works fine
> class(us.states@data)
[1] "data.table" "data.frame"

us.states@data[,test:=1L]

Warning produced from this line:

Warning message: In[.data.table(us.states@data, , :=(test, 1L)) : Invalid .internal.selfref detected and fixed by taking a (shallow) copy of the data.table so that := can add this new column by reference. At an earlier point, this data.table has been copied by R (or been created manually using structure() or similar). Avoid key<-, names<- and attr<- which in R currently (and oddly) may copy the whole data.table. Use set* syntax instead to avoid copying: ?set, ?setnames and ?setattr. Also, in R<=v3.0.2, list(DT1,DT2) copied the entire DT1 and DT2 (R's list() used to copy named objects); please upgrade to R>v3.0.2 if that is biting. If this message doesn't help, please report to datatable-help so the root cause can be fixed.

Further, no update was made:

> names(us.states@data)
[1] "STATEFP"  "STATENS"  "AFFGEOID" "GEOID"    "STUSPS"   "NAME"  
    "LSAD"     "ALAND"    "AWATER"

The problem appears to be similar to the general .internal.selfref problem alluded to in several SO answers--I'll choose this one as canon--namely, that the data for a SPDF is stored as an element of a list.

Not clear how to get around this without creating a copy.

Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.