Warn if `base::scale()` is used?

Open
#5,202 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
r
Domain
data, performance

Research direction

Start by reviewing the issue's benchmark comparing base::scale(), data.table usage, and the custom scale_fun in R. Determine whether warning about base::scale() is appropriate and define the expected behavior; the payload names no files or tests, so the implementation entry point is not yet specified.

Written by the indexing model from the issue text.

Description

performance

I suppose base::scale() is often used by users to scale data (1, 2, 3), but it's actually slow. It seems to me that a better solution is to define a custom function for scaling and use internal optimizations from {data.table}. Do you think a warning if base::scale() is used would be reasonable?

library("data.table"); setDTthreads(1)

set.seed(123)
mat = matrix(c(rnorm(1e7, 30, 0.2), runif(1e7, 3, 5), runif(1e7, 10, 20)),
             ncol = 3)
dt = data.table(mat)
cols = colnames(dt)
scale_fun = function(x) {(x - mean(x)) / sd(x)}

result = bench::mark(
  iterations = 10, check = FALSE, time_unit = "s",
  base = base::scale(mat),
  dt_base = dt[, (cols) := lapply(.SD, function(x) as.vector(scale(x))),
                .SDcols = cols],
  dt = dt[, (cols) := lapply(.SD, scale_fun), .SDcols = cols]
)

result
#>   expression   min median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr> <dbl>  <dbl>     <dbl> <bch:byt>    <dbl>
#> 1 base       3.04   3.21      0.314    2.35GB    1.44
#> 2 dt_base    3.25   3.40      0.295     2.8GB    1.53
#> 3 dt         0.397  0.406     2.20   228.97MB    0.88
Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.