tidymodels / tidymodels/workflows

warning for novel levels appear even if you use `step_novel()`

Open
#112 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
R
Stars
211
Forks
26
Avg merge
1h 58m
Merged PRs (30d)
1

Description

If you use step_novel() in a workflow, you can get the warning Warning: Novel levels found in column 'x': '3'. The levels have been removed, and values have been coerced to 'NA'., which seems like a mistake since step_novel() should be able to take care of it.

I think the warning comes from hardhat::scream().

library(tidymodels)

data <-
  data.frame(
    y = as.factor(c(0, 1, 0, 0, 0)),
    x = as.factor(c(0, 1, 2, 0, 0))
  )

test <-
  data.frame(
    x = as.factor(c(0, 1, 2 ,3))
  )

model_spec <- decision_tree() %>%
  set_engine("rpart") %>%
  set_mode("classification")

rec_spec <- 
  recipe(y ~ x, data = data) %>%
  step_novel(x)

rf_workflow <- 
  workflow() %>%
  add_recipe(rec_spec) %>%
  add_model(model_spec)

final_model <- fit(rf_workflow, data)

test_predict <- predict(final_model, test)
#> Warning: Novel levels found in column 'x': '3'. The levels have been removed,
#> and values have been coerced to 'NA'.

test_predict
#> # A tibble: 4 x 1
#>   .pred_class
#>   <fct>      
#> 1 0          
#> 2 0          
#> 3 0          
#> 4 0

Created on 2021-07-04 by the reprex package (v2.0.0)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the warning with the workflow and recipe shown in the issue, then trace the prediction path through hardhat::scream(). Confirm how step_novel() handles the unseen level and whether the warning is still appropriate; done means the example no longer emits a misleading warning while prediction remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
40/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.