scikit-learn / scikit-learn/scikit-learn
inconsistent treatment of None and np.NaN in SimpleImputer
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 67.3k
- Forks
- 27.4k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 58
Description
Doing constant imputation treats only the "missing_value" as missing, so a None by default stays there:
from sklearn.impute import SimpleImputer
import numpy as np
X = np.array([1, 2, np.NaN, None]).reshape(-1, 1)
SimpleImputer(strategy='constant', fill_value="asdf").fit_transform()
array([[1],
[2],
['asdf'],
[None]], dtype=object)
However, using strategy='mean' coerces the None to NaN and so both are replaced:
SimpleImputer(strategy='mean').fit_transform(X)
array([[1. ],
[2. ],
[1.5],
[1.5]])
I don't think the definition of what's missing should depend on the strategy. @thomasjpfan argues that the current constant behavior is inconvenient because it means you have to impute both values separately if you want to one-hot-encode.
It seems more safe to treat them differently but I'm not sure there's a use-case for that. This came up in #17317.
I think this only matters in these two, as other imputers don't allow dtype object arrays.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the SimpleImputer implementation and its existing tests for object arrays, then reproduce the examples with strategy='constant' and strategy='mean'. Review the discussion around #17317 and determine the intended treatment of None and np.NaN; done means the behavior is consistent with that decision and covered by regression tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100