tidymodels / tidymodels/rsample
Proposal: Extend sampling functions by calibration sampling methods
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 343
- Forks
- 67
- Avg merge
- 1h 9m
- Merged PRs (30d)
- 2
Description
I often find myself in data-modeling situations where the existing functions in rsample for setting up a proper assessment/analysis or test/train do not suffice.
Example: A multivariate regression problem, where numeric predictor data distributions are very frequent and centered around a region and only fewer observation are more distant, while the intention is to learn on all data especially effects when moving outside those frequent centered regions.
The risk of just learning the effect in the center by sampling sampling randomly test/train or assessment/analsis or even with some univariate stratification is high.Also the risk of getting inconsistent model performance results is higher.
I suggest to add functionality to rsample which has extended capability for sampling for these cases:
They ensure maximum coverage of data space for both test/train, resp. Assessment/analysis.
The problem is adressed by calibration sampling methods:
Have a look here for some:
https://cran.r-project.org/web/packages/prospectr/vignettes/prospectr.html#duplex-duplex
Literature:
- Snee, R. D. (1977). Validation of regression models: methods and examples. Technometrics, 19(4), 415-428.
- https://delwende.github.io/thesis-final.pdf#page12
- https://www.scielo.br/j/cr/a/9SCp8CFXPRVtWgZHCxnSGwj/?format=html&lang=en
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No implementation files, tests, or entry points are named. Start by reviewing the existing rsample functions for analysis/test and training/assessment splits, then study the linked prospectr calibration-sampling vignette and references. Done would require an agreed API and calibration methods that improve coverage across the data space.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100