performance of dcast could be improved

Open
#6,361 0 comments 3 reactions 1 assignee View on GitHub

@tdhock is already working on this.

Since Aug 8, 2024.

Assessment

This issue has not been assessed yet.

Description

performance reshape

I discovered that there is room to improve the computation time of dcast.
Here is an example:

library(data.table)
num.threads <- 1
setDTthreads(num.threads)
w.res <- atime::atime(
  N=2^seq(1,50),
  setup={
    (row.id.vec <- 1+(seq(0,N-1) %% nrow(iris)))
    N.dt <- melt(data.table(iris[row.id.vec,])[,orig.row.i:=.I],measure=1:4)
  },
  seconds.limit=1,
  dcast=dcast(N.dt, orig.row.i ~ variable),
  pivot=collapse::pivot(N.dt, how="w", ids="orig.row.i", values="value", names="variable", nthreads=num.threads))
w.refs <- atime::references_best(w.res)
w.pred <- predict(w.refs)
N.at.limit <- dcast(w.pred$prediction, . ~ expr.name, value.var="N")
library(ggplot2)
plot(w.pred)+ggtitle(
  sprintf("pivot %.1fx faster than dcast", N.at.limit[, pivot/dcast]))

The code above uses melt to construct N.dt which is a table of size N *4 (where 4=number of numeric columns in iris data), a long version of iris data.
Then dcast is used to return the long iris data into their original/wide shape.
This is a simple operation which just involves allocating a new wide table and copying the values from the long table to the wide table.
On my computer I got the result below.
image
The figure shows that collapse::pivot is faster than dcast for all N, and about ~10x faster asymptotically.

Note that this observation (10x speed difference) only applies to the simple case of copying values into a new shape (no aggregation).
In the case with fun.aggregate, performance looks much more reasonable, see my recent blog post https://tdhock.github.io/blog/2024/collapse-reshape/ for full details.

Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.