tidymodels / tidymodels/probably

better detection of groups/tuning parameters

Open
#92 2 comments 0 reactions 1 assignee View on GitHub

@EmilHvitfeldt is already working on this.

Since May 3, 2023.

bug
Dominant language
R
Stars
123
Forks
16
PR merge metrics
No merged PRs in 30d

Description

When using data frames generated from the tune_*() functions, we silently produce a single plot/analysis if the user doesn't correctly specify what they want.

We should detect this (when there is more than one config) and produce a meaningful error.

Also, the plot functions have a group argument and the estimation functions require group_by(). That's confusing.

Example:

library(tidymodels)
library(probably)
#> 
#> Attaching package: 'probably'
#> The following objects are masked from 'package:base':
#> 
#>     as.factor, as.ordered
library(bonsai)

tidymodels_prefer()
theme_set(theme_bw())
options(pillar.advice = FALSE, pillar.min_title_chars = Inf)

set.seed(1345)
cls_train <- sim_classification(1000)
cls_test  <- sim_classification( 500)
cls_calib <- sim_classification( 500)

set.seed(7378)
cls_rs <- vfold_cv(cls_train)

lgb_spec <- boost_tree() %>% set_mode("classification") %>% set_engine("lightgbm")

cls_metrics <- metric_set(brier_class, roc_auc)

set.seed(6929)
lgb_tune_res <-
  boost_tree(min_n = tune()) %>%
  set_mode("classification") %>%
  set_engine("lightgbm") %>%
  tune_grid(
    class ~ .,
    resamples = cls_rs,
    control = control_resamples(save_pred = TRUE),
    metrics = cls_metrics,
    grid = tibble(min_n = c(2, 50))
  )

df_pred_res  <- lgb_res %>% collect_predictions()
#> Error in collect_predictions(.): object 'lgb_res' not found
df_pred_tune_res  <- lgb_tune_res %>% collect_predictions()

df_new <- df_pred_res[1:5,]
#> Error in eval(expr, envir, enclos): object 'df_pred_res' not found
df_tune_new <- df_pred_tune_res %>% dplyr::slice(1:5, .by = .config)
# Plotting issues

# This produces 1 plot; should be two
df_pred_tune_res %>%
  cal_plot_windowed(truth = class, estimate = .pred_class_1,
                    window_size = 0.1, step_size = 0.025)


# Using `group` makes two plots
df_pred_tune_res %>%
  cal_plot_windowed(truth = class, estimate = .pred_class_1, group = .config,
                    window_size = 0.1, step_size = 0.025)

# Estimation issues

# Should have two groups
df_pred_tune_res %>%
  cal_estimate_logistic(truth = class)
#> 
#> ── Probability Calibration
#> Method: Logistic Spline
#> Type: Binary
#> Source class: Data Frame
#> Data points: 2,000
#> Truth variable: `class`
#> Estimate variables:
#> `.pred_class_1` ==> class_1
#> `.pred_class_2` ==> class_2

# Has two groups via a different "by" mechanism:
df_pred_tune_res %>%
  group_by(.config) %>% 
  cal_estimate_logistic(truth = class)
#> 
#> ── Probability Calibration
#> Method: Logistic Spline
#> Type: Binary
#> Source class: Data Frame
#> Data points: 2,000, split in 2 groups
#> Truth variable: `class`
#> Estimate variables:
#> `.pred_class_1` ==> class_1
#> `.pred_class_2` ==> class_2

Created on 2023-03-21 by the reprex package (v2.0.1)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.