tidymodels / tidymodels/rsample
Nested CV is Memory-inefficient
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 343
- Forks
- 67
- Avg merge
- 1h 9m
- Merged PRs (30d)
- 2
Description
Hello,
I noticed that when using nested_cv (or bringing over a custom nested resampling scheme via caret2rsample(), the memory does scale linearly with the number of folds. This is in contrast to a single resampling (for example the bootstrap resampling example on the rsample documentation landing page), where a feature of the package is that the memory doesn't blow up when resampling a dataset many times.
I adapted that example to show what I'm talking about:
library(rsample)
library(mlbench)
data(LetterRecognition)
lobstr::obj_size(LetterRecognition)
# 2,644,640 B
set.seed(35222)
#For an example, an outer 5 fold cv with each outer fold having an inner 2 fold cv
nested <- rsample::nested_cv(LetterRecognition, outside = vfold_cv(times = 5),inside=vfold_cv(times = 2))
lobstr::obj_size(nested)
#34,434,200 B
# Object size per resample - Actually slightly bigger than what we started with
lobstr::obj_size(nested)/nrow(nested)
#3,443,420 B
# Fold increase is > 10
as.numeric(lobstr::obj_size(nested)/lobstr::obj_size(LetterRecognition))
#13.02037
Unless I'm missing something, nested resampling on a large dataset doesn't seem to be possible, since the memory required will quickly add up.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the nested_cv() example using LetterRecognition, vfold_cv(), and lobstr::obj_size(), then compare it with the bootstrap resampling example linked in the issue. Investigate how nested resamples retain data and verify the result by measuring object size as the number of folds changes; done means nested resampling no longer scales linearly in memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100