scikit-learn / scikit-learn/scikit-learn

inconsistent treatment of None and np.NaN in SimpleImputer

Open
#17,625 8 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug
Dominant language
Python
Stars
67.3k
Forks
27.4k
Avg merge
1d 15h
Merged PRs (30d)
58

Description

Doing constant imputation treats only the "missing_value" as missing, so a None by default stays there:

from sklearn.impute import SimpleImputer
import numpy as np
X = np.array([1, 2, np.NaN, None]).reshape(-1, 1)
SimpleImputer(strategy='constant', fill_value="asdf").fit_transform()
array([[1],
       [2],
       ['asdf'],
       [None]], dtype=object)

However, using strategy='mean' coerces the None to NaN and so both are replaced:

SimpleImputer(strategy='mean').fit_transform(X)
array([[1. ],
       [2. ],
       [1.5],
       [1.5]])

I don't think the definition of what's missing should depend on the strategy. @thomasjpfan argues that the current constant behavior is inconvenient because it means you have to impute both values separately if you want to one-hot-encode.

It seems more safe to treat them differently but I'm not sure there's a use-case for that. This came up in #17317.
I think this only matters in these two, as other imputers don't allow dtype object arrays.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the SimpleImputer implementation and its existing tests for object arrays, then reproduce the examples with strategy='constant' and strategy='mean'. Review the discussion around #17317 and determine the intended treatment of None and np.NaN; done means the behavior is consistent with that decision and covered by regression tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.