tidymodels / tidymodels/workflows
Retain calibration data
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 211
- Forks
- 26
- Avg merge
- 1h 58m
- Merged PRs (30d)
- 1
Description
add_calibration_data() would solve the problems of
- Having the calibration data being peeled off of the training/analysis set
- Users are unable to use a static calibration set.
- More reproducible workflows (see tidymodels/tune#1038) and easier debugging.
- Enable more advanced postprocessors for different cases (forecasting, applicability domains, etc.)
When using add_calibration_data(), a tibble that conforms to the training set mold is required.
The calibration set would be stored in worflow$pre$calibration (similar to case weights).
When fit.workflow() or .fit_post() are called, they first preprocess the calibration set (if any), make predictions, and pass those predictions* to the calibrator.
*I believe that we should also have a way to pass in the molded calibration set for the trailer adjustments. We have 2-3 cases where those are required.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the proposed add_calibration_data() entry point and the fit.workflow() and .fit_post() call paths. Trace how workflow$pre currently stores case weights and how a training-set mold is represented. Done should include a decided design for retaining, preprocessing, and passing calibration data to calibrators, including molded data needed for trailer adjustments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100