scikit-learn / scikit-learn/scikit-learn
Allow 2d array input in CountVectorizer?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 67.3k
- Forks
- 27.4k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 58
Description
I wonder if we should allow (n_samples, 1) input to CountVectorizer. I think we had this discussion before, right now we basically have a special case in ColumnTransformer with scalar columns to produce 1d arrays.
Does anyone remember where this discussion was?
A lot of my students are struggling with putting a CountVectorizer into a ColumnTransformer and it might be nice to be more generous with the input formats, and ensure we raise nice error messages.
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
X = pd.DataFrame(["what is my purpose?", "You bring the butter."])
CountVectorizer().fit_transform(X)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-6-6cae33e8c706> in <module>
3 from sklearn.compose import ColumnTransformer
4 X = pd.DataFrame(["what is my purpose?", "You bring the butter."])
----> 5 CountVectorizer().fit_transform(X)
~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in fit_transform(self, raw_documents, y)
1218
1219 vocabulary, X = self._count_vocab(raw_documents,
-> 1220 self.fixed_vocabulary_)
1221
1222 if self.binary:
~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _count_vocab(self, raw_documents, fixed_vocab)
1129 for doc in raw_documents:
1130 feature_counter = {}
-> 1131 for feature in analyze(doc):
1132 try:
1133 feature_idx = vocabulary[feature]
~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _analyze(doc, analyzer, tokenizer, ngrams, preprocessor, decoder, stop_words)
101 else:
102 if preprocessor is not None:
--> 103 doc = preprocessor(doc)
104 if tokenizer is not None:
105 doc = tokenizer(doc)
~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _preprocess(doc, accent_function, lower)
66 """
67 if lower:
---> 68 doc = doc.lower()
69 if accent_function is not None:
70 doc = accent_function(doc)
AttributeError: 'int' object has no attribute 'lower'
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with CountVectorizer.fit_transform and _count_vocab in the feature-extraction text implementation, then compare the scalar-column handling in ColumnTransformer. Clarify which (n_samples, 1) inputs are supported and what errors should be reported, and add regression coverage for the accepted and rejected forms.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100