tidymodels / tidymodels/probably
better detection of groups/tuning parameters
Open
@EmilHvitfeldt is already working on this.
Since May 3, 2023.
bug
- Dominant language
- R
- Stars
- 123
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
When using data frames generated from the tune_*() functions, we silently produce a single plot/analysis if the user doesn't correctly specify what they want.
We should detect this (when there is more than one config) and produce a meaningful error.
Also, the plot functions have a group argument and the estimation functions require group_by(). That's confusing.
Example:
library(tidymodels)
library(probably)
#>
#> Attaching package: 'probably'
#> The following objects are masked from 'package:base':
#>
#> as.factor, as.ordered
library(bonsai)
tidymodels_prefer()
theme_set(theme_bw())
options(pillar.advice = FALSE, pillar.min_title_chars = Inf)
set.seed(1345)
cls_train <- sim_classification(1000)
cls_test <- sim_classification( 500)
cls_calib <- sim_classification( 500)
set.seed(7378)
cls_rs <- vfold_cv(cls_train)
lgb_spec <- boost_tree() %>% set_mode("classification") %>% set_engine("lightgbm")
cls_metrics <- metric_set(brier_class, roc_auc)
set.seed(6929)
lgb_tune_res <-
boost_tree(min_n = tune()) %>%
set_mode("classification") %>%
set_engine("lightgbm") %>%
tune_grid(
class ~ .,
resamples = cls_rs,
control = control_resamples(save_pred = TRUE),
metrics = cls_metrics,
grid = tibble(min_n = c(2, 50))
)
df_pred_res <- lgb_res %>% collect_predictions()
#> Error in collect_predictions(.): object 'lgb_res' not found
df_pred_tune_res <- lgb_tune_res %>% collect_predictions()
df_new <- df_pred_res[1:5,]
#> Error in eval(expr, envir, enclos): object 'df_pred_res' not found
df_tune_new <- df_pred_tune_res %>% dplyr::slice(1:5, .by = .config)
# Plotting issues
# This produces 1 plot; should be two
df_pred_tune_res %>%
cal_plot_windowed(truth = class, estimate = .pred_class_1,
window_size = 0.1, step_size = 0.025)

# Using `group` makes two plots
df_pred_tune_res %>%
cal_plot_windowed(truth = class, estimate = .pred_class_1, group = .config,
window_size = 0.1, step_size = 0.025)

# Estimation issues
# Should have two groups
df_pred_tune_res %>%
cal_estimate_logistic(truth = class)
#>
#> ── Probability Calibration
#> Method: Logistic Spline
#> Type: Binary
#> Source class: Data Frame
#> Data points: 2,000
#> Truth variable: `class`
#> Estimate variables:
#> `.pred_class_1` ==> class_1
#> `.pred_class_2` ==> class_2
# Has two groups via a different "by" mechanism:
df_pred_tune_res %>%
group_by(.config) %>%
cal_estimate_logistic(truth = class)
#>
#> ── Probability Calibration
#> Method: Logistic Spline
#> Type: Binary
#> Source class: Data Frame
#> Data points: 2,000, split in 2 groups
#> Truth variable: `class`
#> Estimate variables:
#> `.pred_class_1` ==> class_1
#> `.pred_class_2` ==> class_2
Created on 2023-03-21 by the reprex package (v2.0.1)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.