tidymodels / tidymodels/probably
Generalize `int_conformal_quantile()`
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 123
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
Currently int_conformal_quantile() seems limited in that it:
- doesn't work for arbitrary model types
- doesn't use components of the workflow, etc. in producing the calibrated quantiles (e.g. the recipe, or the holdout samples if doing resampling,...)
- may be inconsistent with point estimate from the base model
Ideally, for any model/workflow fit that is set-up to output quantiles (or intervals), int_conformal_quantile() would simply use the calibration data (or the available held-out data like int_conformal_cv() does if set-up resamples) to adjust the quantiles outputted by the fitted workflow.
As described from 31:00 to 37:00 by Angelopuoulos and Bates here: https://www.youtube.com/watch?v=nql000Lu_iE&list=PLXs7Va5fWFZ72DTVcx4qIvny1xNrl68PK&index=1), the steps then would be: (with parsnip / workflows) train an arbitrary model (that is capable of optimizing on pinball loss function / outputting quantiles / intervals) --> pass the resulting object into (a generalized version of) int_conformal_quantile() whose responsibility it would be to calibrate the quantiles from the model/workflow (which would be a similar set-up but that just doesn't have the probably:::quant_train() step so more similar to how the other int_conformal_*() functions work) --> which could then be used to produce well-calibrated intervals on new data.
I imagine this would be dependent on integrated support in parsnip for quantiles (https://github.com/tidymodels/parsnip/issues/119, https://github.com/tidymodels/parsnip/issues/465). Figured may as well open an issue though.
Rough ex with a ranger workflow:
### Set-up a ranger workflow (that has a recipe) for quantile regression forests
library(tidyverse)
library(tidymodels)
library(AmesHousing)
ames <- make_ames() %>%
mutate(Years_Old = Year_Sold - Year_Built,
Years_Old = ifelse(Years_Old < 0, 0, Years_Old))
set.seed(4595)
data_split <- initial_split(ames, strata = "Sale_Price", prob = 0.6)
ames_train <- training(data_split)
ames_holdout <- testing(data_split)
rf_recipe <-
recipe(
Sale_Price ~ Lot_Area + Neighborhood + Years_Old + Gr_Liv_Area + Overall_Qual + Total_Bsmt_SF + Garage_Area,
data = ames_train
) %>%
step_log(Sale_Price, base = 10) %>%
step_other(Neighborhood, Overall_Qual, threshold = 50) %>%
step_novel(Neighborhood, Overall_Qual) %>%
step_dummy(Neighborhood, Overall_Qual)
rf_mod <- rand_forest() %>%
set_engine("ranger", importance = "impurity", seed = 63233, quantreg = TRUE) %>%
set_mode("regression")
set.seed(63233)
rf_wf <- workflows::workflow() %>%
add_model(rf_mod) %>%
add_recipe(rf_recipe) %>%
fit(ames_train)
#### Conformalize quantiles from model/workflow
set.seed(1243)
val_new_split <- initial_split(ames_holdout, prop = 0.50)
ames_cal <- training(val_new_split)
ames_new <- testing(val_new_split)
quant_predict <- function(fit, new_data, level) {
alpha <- (1 - level)
quant_pred <- predict(fit, new_data, type = "quantiles", quantiles = c(alpha / 2, 1 - (alpha / 2)))
quant_pred <- dplyr::as_tibble(quant_pred)
quant_pred <- stats::setNames(quant_pred, c(".pred_lower", ".pred_upper"))
quant_pred
}
level <- 0.90
cal_data_baked <- workflows::extract_recipe(rf_wf) %>% bake(ames_cal)
new_data_baked <- workflows::extract_recipe(rf_wf) %>% bake(ames_new)
preds_q <- bind_cols(
select(cal_data_baked, Sale_Price),
quant_predict(rf_wf$fit$fit$fit, cal_data_baked, level = level)
)
resid <- preds_q %>%
mutate(R_low = .pred_lower - Sale_Price,
R_high = Sale_Price - .pred_upper) %>%
with(pmax(R_low, R_high))
q_hat <- quantile(resid, probs = level)
preds_q_new <- bind_cols(
select(new_data_baked, Sale_Price),
quant_predict(rf_wf$fit$fit$fit, new_data_baked, level = 0.90)
) %>%
mutate(.pred_lower = .pred_lower - q_hat,
.pred_upper = .pred_upper + q_hat)
preds_q_new %>%
summarise(coverage = mean(Sale_Price <= .pred_upper & Sale_Price >= .pred_lower))
The adaptability of the intervals then would be coming from the model in the workflow being able to output quantiles / intervals (rather than from overriding the workflow and retraining for the interval). Even if the underlying workflow isn't that adaptive (e.g. say the user has a workflow for an lm model that is just returning standard prediction intervals based on variance) the approach described above would likely do a slightly better job at factoring in the epistemic uncertainty in the model estimation compared to just doing int_conformal_split(), because it would allow for wider intervals further from the data centroid, which doesn't happen with int_conformal_split().
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading int_conformal_quantile() and int_conformal_cv() to compare their current handling of fitted models, workflows, calibration data, and resampling. Review the linked parsnip quantile-support issues and the ranger workflow example. Done means quantile-capable model or workflow fits can be calibrated using available calibration or held-out data without retraining the workflow for intervals.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100