dask / dask/dask-ml

HashingVectorizer behaves differently from FeatureHasher

Open
#963 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:
[HashingVectorizer](https://ml.dask.org/modules/generated/dask_ml.feature_extraction.text.HashingVectorizer.html#dask-ml-feature-extraction-text-hashingvectorizer) behaves differently from FeatureHasher, HashingVectorizer can work off a Sting like

```
JUNK_FOOD_DOCS = (
"the pizza pizza beer copyright",
"the pizza burger beer copyright",
"the the pizza beer beer copyright",
"the burger beer beer copyright",
"the coke burger coke copyright",
"the coke burger burger",
)

```

but [FeatureHasher](https://ml.dask.org/modules/generated/dask_ml.feature_extraction.text.FeatureHasher.html#dask-ml-feature-extraction-text-featurehasher) expects an iterable of strings like:

```
JUNK_FOOD_DOCS = [["the", "pizza", "pizza", "beer", "copyright"],
["the", "coke", "burger", "burger"] ]

```
Which is the correct behavior:

1. expect the hasher to parse strings into vectors
2. fix the test by sending list of lists of strings to FeatureHasher instead list of strings, like the other hasher expects

**Minimal Complete Verifiable Example**:

[See dask-ml/tests/feature_extraction/test_text.py: test_basic() ](https://github.com/dask/dask-ml/blob/ba2024d68557fade43f4dc2fa369445d01629923/tests/feature_extraction/test_text.py#L33)

**Anything else we need to know?**:

This is illustrated in the [failing test](https://github.com/dask/dask-ml/actions/runs/4083895663/jobs/7039952888#step:5:817)

**Environment**:

- Dask version: dask-ml-3.8 conda env dask 2023.1.1
- Python version: 3.8, 3.9, 3.10
- Operating System: ubuntu-latest
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.