separate index attributes from index

Open
#4,467 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Stale
Tech stack
r
Domain
performance

Research direction

Start with forderv(retGrp=TRUE), the index attribute, and the setindexv context referenced in the issue. Read the linked data.table#4386 commit to understand how index reuse depends on the current attribute structure. Done means the index and its extra attributes have a separate, consistent representation without the reported shallow-duplication overhead.

Written by the indexing model from the issue text.

Description

High internals performance question

Currently when calling forderv(retGrp=TRUE) the extra attributes are set on integer vector, on the index.

d = data.table(x=c(2L,2:1))
#setindexv(d, "x") ## if you are on #4386
setattr(d, "index", setattr(integer(), "__x", forderv(d, "x", retGrp=TRUE)))

I would like to propose to wrap the index into list, and set extra attributes on a list rather than on the index. Or just keep attributes in the same list.

Currently:

str(attr(attr(d, "index"), "__x"))
# int [1:3] 3 1 2
# - attr(*, "starts")= int [1:2] 1 2
# - attr(*, "maxgrpn")= int 2

proposed, either of those, latter one seems to be easier to operate on:

str(attr(attr(d, "index"), "__x"))
#List of 1
# $ : int [1:3] 3 1 2
# - attr(*, "starts")= int [1:2] 1 2
# - attr(*, "maxgrpn")= int 2

str(attr(attr(d, "index"), "__x"))
#List of 3
# $ index  : int [1:3] 3 1 2
# $ starts : int [1:2] 1 2
# $ maxgrpn: int 2

Motivation to change this internal structure is that, AFAIU, as of now, we cannot shallow duplicate index integer vector alone, that adds relatively big overhead if we want to re-use the index, but we don't want to have extra attributes on it.
Exact use case for that is described in https://github.com/Rdatatable/data.table/pull/4386/commits/1606046476d20a806fff6082bdd15d909eac0285
For an index of length 1e8 we are already losing 0.258s just to re-use existing index.

Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.