scikit-learn / scikit-learn/scikit-learn

Allow 2d array input in CountVectorizer?

Open
#16,972 5 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Enhancement module:feature_extraction
Dominant language
Python
Stars
67.3k
Forks
27.4k
Avg merge
1d 15h
Merged PRs (30d)
58

Description

I wonder if we should allow (n_samples, 1) input to CountVectorizer. I think we had this discussion before, right now we basically have a special case in ColumnTransformer with scalar columns to produce 1d arrays.
Does anyone remember where this discussion was?

A lot of my students are struggling with putting a CountVectorizer into a ColumnTransformer and it might be nice to be more generous with the input formats, and ensure we raise nice error messages.

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
X = pd.DataFrame(["what is my purpose?", "You bring the butter."])
CountVectorizer().fit_transform(X)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-6-6cae33e8c706> in <module>
      3 from sklearn.compose import ColumnTransformer
      4 X = pd.DataFrame(["what is my purpose?", "You bring the butter."])
----> 5 CountVectorizer().fit_transform(X)

~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in fit_transform(self, raw_documents, y)
   1218 
   1219         vocabulary, X = self._count_vocab(raw_documents,
-> 1220                                           self.fixed_vocabulary_)
   1221 
   1222         if self.binary:

~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _count_vocab(self, raw_documents, fixed_vocab)
   1129         for doc in raw_documents:
   1130             feature_counter = {}
-> 1131             for feature in analyze(doc):
   1132                 try:
   1133                     feature_idx = vocabulary[feature]

~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _analyze(doc, analyzer, tokenizer, ngrams, preprocessor, decoder, stop_words)
    101     else:
    102         if preprocessor is not None:
--> 103             doc = preprocessor(doc)
    104         if tokenizer is not None:
    105             doc = tokenizer(doc)

~/miniconda3/lib/python3.7/site-packages/sklearn/feature_extraction/text.py in _preprocess(doc, accent_function, lower)
     66     """
     67     if lower:
---> 68         doc = doc.lower()
     69     if accent_function is not None:
     70         doc = accent_function(doc)

AttributeError: 'int' object has no attribute 'lower'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CountVectorizer.fit_transform and _count_vocab in the feature-extraction text implementation, then compare the scalar-column handling in ColumnTransformer. Clarify which (n_samples, 1) inputs are supported and what errors should be reported, and add regression coverage for the accepted and rejected forms.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.