dmlc / dmlc/xgboost

Unseen Categories Using Categorical Re-coder

Open
#11,826 14 comments 0 reactions 0 assignees View on GitHub
status: need update
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

I’m using the latest version (3.1.1) and testing the new Categorical Re-coder feature in Python. During prediction, I encountered the following error:

Found a category not in the training set.

In previous versions, I handled this by using pd.Categorical and manually storing the categories during training. At inference time, any unseen category was automatically converted to NaN, which allowed the model to process it without failing.

With the new Categorical Re-coder in 3.1, if an unseen category appears during inference, the process fails with the above error.

Is this the intended behavior for the new Categorical Re-coder?
If so, what is the recommended approach for handling unseen categories during inference?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the prediction failure with XGBoost 3.1.1, the new Categorical Re-coder, and an inference category absent from training. Compare this with the earlier pd.Categorical workflow described in the issue. Done means establishing whether unseen categories should be accepted and documenting or implementing the recommended inference behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.