tidymodels / tidymodels/rsample

Feature request: combining/crossing multiple resampling methods

Open
#455 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
R
Stars
343
Forks
67
Avg merge
1h 9m
Merged PRs (30d)
2

Description

Here are few scenarios I would like to describe in this feature request:

  1. Combining/crossing multiple resampling methods (e.g. spatial_nndm_cv() crossed with sliding_period() resamples)

  2. Determining/benchmarking best model similar to rank_results() for different resampling methods (e.g. group_vfold_cv(cities) vs. spatial_clustering_cv(coords = (x, y))

  3. Ability to input a list of multiple resampling methods into model specifications e.g. workflow_set(resamples = list(*)), similar to passing workflow_set(models = list(*), preproc = list(*)) (or ability to pass a resamples list akin to workflow_map(resamples = list(*)))

Here are some example questions -- would spatial_clustering_cv(coords = (x, y) be better or would group_vfold_cv(cities) be better for model tuning? How much would granularity matter in model tuning e.g. group_vfold_cv(cities) vs. group_vfold_cv(municipalities)? How much is the model effected by resampling efficiency (e.g. vs. bootstraps)?

An example decision making point could be that if there is small accuracy loss difference but big run time (or electric costs) difference between cities vs. province, then province would be selected. The results could be shown from the output of rank_results(), with a column showing which resampling was used.

Then for utility, saving this model with a particular resampling method so this model specification can be reused right away again would be nice (would this be tidypredict_fit()? augment()? bake()?).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the resampling methods named in the request, including spatial_nndm_cv(), sliding_period(), group_vfold_cv(), spatial_clustering_cv(), and the workflow_set(), workflow_map(), and rank_results() entry points. Define how multiple resampling methods would be combined, compared, and represented in results, then document the expected behavior and reuse workflow before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.