dask / dask/dask-ml

Default datatype when using CountVectorizer and HashingVectorizer should be sparse COO

Open
#952 0 comments 0 reactions 0 assignees View on GitHub
needs triage
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

As mentioned in this issue [here](https://github.com/dask/dask/issues/4664#issuecomment-479170810_) scipy sparse matrix class has little to no functions available from the Dask Array module. For example, raising `TTypeError: _cs_matrix.sum() got an unexpected keyword argument 'keepdims'` when using `da.sum(sparse_matrix, axis=0)` or `sparse_matrix.sum()`

When using CountVectorizer and HashingVectorizer both return blocks of scipy.sparse_csr.csr_matrix data type.

To interact with those blocks, one has to do a change with the sparse COO module. See https://docs.dask.org/en/latest/array-sparse.html this fixed a problem i was having and trying to correct for multiple hours.

Default datatype should be sparse_coo.core.COO even at the cost if increasing depedencies, due to the fact that the result would be more dask-like and managaeable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.