tidymodels / tidymodels/probably

add a seed argument to some adjustments

Open
#181 1 comment 0 reactions 1 assignee View on GitHub

@EmilHvitfeldt is already working on this.

Since Apr 29, 2025.

feature
Dominant language
R
Stars
123
Forks
16
PR merge metrics
No merged PRs in 30d

Description

Working with adjust_probability_calibration() and I get slightly different calibration results across tuning parameters:

# A tibble: 9 × 7
  threshold .metric     .estimator   mean     n std_err .config        
      <dbl> <chr>       <chr>       <dbl> <int>   <dbl> <chr>          
1       0   roc_auc     binary     0.712     10 0.0216  pre0_mod0_post1
2       0   sensitivity binary     1         10 0       pre0_mod0_post1
3       0   specificity binary     0         10 0       pre0_mod0_post1
4       0.5 roc_auc     binary     0.710     10 0.0212  pre0_mod0_post2
5       0.5 sensitivity binary     0.195     10 0.0213  pre0_mod0_post2
6       0.5 specificity binary     0.969     10 0.00612 pre0_mod0_post2
7       1   roc_auc     binary     0.710     10 0.0229  pre0_mod0_post3
8       1   sensitivity binary     0.0248    10 0.0184  pre0_mod0_post3
9       1   specificity binary     0.996     10 0.00297 pre0_mod0_post3

This isn't a big deal, but it could confuse users.

I suggest adding an argument seed = sample.int(10^4, 1) for adjustments that use random numbers. This will be evaluated when the tailor is made and using withr we can fix the stream when the adjustment is trained.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.