scikit-learn / scikit-learn/scikit-learn
Output dimension consistency between PCA and LinearDiscriminantAnalysis
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 67.3k
- Forks
- 27.4k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 58
Description
Description
This follows #11526, regarding some sometimes unexpected behaviour: that LDA can sometimes truncate the scalings_ (and the output of transform), overwriding in a sense what the user has specified in n_components. This time this tackles the case where it happens because the input is made of collinear vectors. The behaviour of PCA and LinearDiscriminantAnalysis is not the same regarding collinear input: PCA (and all other sklearn.decomposition algorithms) transforms the input with some always zero features, but raises no Warning, whereas LDA (with the default solver 'svd') truncates the transform output but raises "Variables are collinear". I wondered if it wouldn't better to uniformize a bit the behaviour, by for instance:
- making PCA raise a warning in that case
- and making LDA transform return a transformation including the zeros (not truncating)
Also maybe the warning "Variables are collinear" could be improved by saying that some dimensions of the output will be zeros.
Steps/Code to Reproduce
Here is an example that shows the different behaviours from LDA and PCA:
Example:
import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.decomposition import PCA
basis = np.random.randn(3, 5) # a basis of rank 3 to generate other vectors
X = np.random.randn(10, 3).dot(basis) # we create 10 collinear points, of n_features=5 and rank is 3
y = np.hstack([1, 1, 2, 2, 3, 3, 4, 4, 5, 6]) # we create a lot of classes so the truncation is due to collinearity and not number of classes
for model, model_name in zip([PCA(), LinearDiscriminantAnalysis()], ['PCA', 'LDA']):
model.set_params(n_components=5)
print('-'*40 + model_name + '-'*40)
model.fit(X, y)
print('transform_shape: %s' % str(model.transform(X).shape))
print((model.transform(X)**2).sum(axis=0)) # sum of squares of features to see features always zero
# print(model.transform(X))
Expected Results
something like that:
----------------------------------------PCA----------------------------------------
transform_shape: (10, 5)
warnings.warn("Variables are collinear. Two features of the output space will always be zero.")
[notzero notzero notzero zero zero]
----------------------------------------LDA----------------------------------------
/home/will/Code/sklearn-forks/wdevazelhes/scikit-learn/sklearn/discriminant_analysis.py:388: UserWarning: Variables are collinear.
warnings.warn("Variables are collinear. Two features of the output space will always be zero.")
transform_shape: (10, 5)
[notzero notzero notzero zero zero]
Actual Results
----------------------------------------PCA----------------------------------------
transform_shape: (10, 5)
[9.32882671e+01 1.86702898e+01 6.05330823e+00 1.30856483e-30
2.75278672e-30]
----------------------------------------LDA----------------------------------------
/home/will/Code/sklearn-forks/wdevazelhes/scikit-learn/sklearn/discriminant_analysis.py:388: UserWarning: Variables are collinear.
warnings.warn("Variables are collinear.")
transform_shape: (10, 3)
[13.9232471 8.3425689 5.96980738]
Versions
Linux-4.4.0-130-generic-x86_64-with-debian-stretch-sid
Python 3.6.3 |Anaconda, Inc.| (default, Nov 3 2017, 19:19:16)
[GCC 7.2.0]
NumPy 1.14.3
SciPy 1.1.0
Scikit-Learn 0.20.dev0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the provided collinear-input example with PCA and LinearDiscriminantAnalysis, then inspect sklearn/discriminant_analysis.py around line 388 and the corresponding PCA implementation. Compare the requested alternatives with existing tests and determine the agreed behavior for output dimensions, zero components, and warning text. Done means the chosen behavior is implemented consistently and covered by regression tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100