tidymodels / tidymodels/rsample

Nested CV is Memory-inefficient

Open
#283 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
R
Stars
343
Forks
67
Avg merge
1h 9m
Merged PRs (30d)
2

Description

Hello,

I noticed that when using nested_cv (or bringing over a custom nested resampling scheme via caret2rsample(), the memory does scale linearly with the number of folds. This is in contrast to a single resampling (for example the bootstrap resampling example on the rsample documentation landing page), where a feature of the package is that the memory doesn't blow up when resampling a dataset many times.

I adapted that example to show what I'm talking about:

library(rsample)
library(mlbench)

data(LetterRecognition)
lobstr::obj_size(LetterRecognition)
# 2,644,640 B

set.seed(35222)
#For an example, an outer 5 fold cv with each outer fold having an inner 2 fold cv
nested <- rsample::nested_cv(LetterRecognition, outside = vfold_cv(times = 5),inside=vfold_cv(times = 2))
lobstr::obj_size(nested)
#34,434,200 B

# Object size per resample - Actually slightly bigger than what we started with
lobstr::obj_size(nested)/nrow(nested)
#3,443,420 B


# Fold increase is > 10
as.numeric(lobstr::obj_size(nested)/lobstr::obj_size(LetterRecognition))
#13.02037

Unless I'm missing something, nested resampling on a large dataset doesn't seem to be possible, since the memory required will quickly add up.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the nested_cv() example using LetterRecognition, vfold_cv(), and lobstr::obj_size(), then compare it with the bootstrap resampling example linked in the issue. Investigate how nested resamples retain data and verify the result by measuring object size as the number of folds changes; done means nested resampling no longer scales linearly in memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.