tidymodels / tidymodels/rsample

Could `make_strata()` warn (or remove the `strata` attribute) when only returning a single strata? Or message when pooling at all?

Open
#441 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
R
Stars
343
Forks
67
Avg merge
1h 9m
Merged PRs (30d)
2

Description

Feature

I was reminded about #438 by the GitHub lock bot, an issue where a user was surprised that vfold_cv() (and eventually make_strata()) "didn't stratify" (or rather, treated the data as only having one stratum) when the stratification variable only had one class above the pooling threshold.

I think rsample is doing the right thing here, and behaving as documented, but this behavior is still a bit surprising. Would it be possible for make_strata() to warn when it only returns a single stratum? I imagine this is almost always unintentional, as users wouldn't specify a stratification variable if they thought it would go unused.

Another consideration here is that, even if only one stratum is created, the rset objects still contain a strata attribute. As a result, when printed these objects claim that they were created "using stratification":

data.frame(
  x = rnorm(100), 
  y = c(rep("a", 99), "b")
) |> 
  rsample::vfold_cv(strata = y)
#> #  10-fold cross-validation using stratification 
#> # A tibble: 10 × 2
#>    splits          id    
#>    <list>          <chr> 
#>  1 <split [90/10]> Fold01
#>  2 <split [90/10]> Fold02
#>  3 <split [90/10]> Fold03
#>  4 <split [90/10]> Fold04
#>  5 <split [90/10]> Fold05
#>  6 <split [90/10]> Fold06
#>  7 <split [90/10]> Fold07
#>  8 <split [90/10]> Fold08
#>  9 <split [90/10]> Fold09
#> 10 <split [90/10]> Fold10

Created on 2023-07-27 with reprex v2.0.2

This might be a bit misleading, as the sampling here didn't depend on the y value at all. Would it make sense to drop the strata attribute if only one stratum is created?

Finally, would it make sense for the categorical branch of make_strata to provide a message listing the categories that get "pooled" together, and which stratum they were pooled into? This might help users catch processing mistakes, if they weren't expecting to have any rare classes that would get automatically pooled. This might be too noisy though, and not as useful as warning about "single stratum" cases.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with make_strata() and its vfold_cv() caller, focusing on how a single stratum, pooled categories, and the strata attribute are represented. Review the existing behavior and tests, then resolve whether completion means warning, removing the attribute, reporting pooled categories, or a documented combination.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.