tidymodels / tidymodels/rsample

Proposal: Extend sampling functions by calibration sampling methods

Open
#600 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
343
Forks
67
Avg merge
1h 9m
Merged PRs (30d)
2

Description

I often find myself in data-modeling situations where the existing functions in rsample for setting up a proper assessment/analysis or test/train do not suffice.

Example: A multivariate regression problem, where numeric predictor data distributions are very frequent and centered around a region and only fewer observation are more distant, while the intention is to learn on all data especially effects when moving outside those frequent centered regions.

The risk of just learning the effect in the center by sampling sampling randomly test/train or assessment/analsis or even with some univariate stratification is high.Also the risk of getting inconsistent model performance results is higher.

I suggest to add functionality to rsample which has extended capability for sampling for these cases:

They ensure maximum coverage of data space for both test/train, resp. Assessment/analysis.

The problem is adressed by calibration sampling methods:
Have a look here for some:

https://cran.r-project.org/web/packages/prospectr/vignettes/prospectr.html#duplex-duplex

Literature:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No implementation files, tests, or entry points are named. Start by reviewing the existing rsample functions for analysis/test and training/assessment splits, then study the linked prospectr calibration-sampling vignette and references. Done would require an agreed API and calibration methods that improve coverage across the data space.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.