“ValueError: cannot reshape array of size ...” from transform
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
First, thanks for putting such a great project together!
I'm having trouble with the transform method. I'm training on TFIDF vectors from a large set of text data (>100k rows, 20k features), of type scipy.sparse.csr.csr_matrix. I'm then applying to new set of test data encoded in the same way. I seem to have some "bad" rows, and get an error if I try to transform the full test set, or an individual "bad" row. That error is:
```
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
in
----> 1 embedding_n.transform(tfidf_test[30103])
/projects/PK13588/CallReason_Exploratory/envs/ffs/lib/python3.6/site-packages/umap/umap_.py in transform(self, X)
2145 # and data. Doing so relies on the constant degree assumption!
2146 csr_graph = normalize(graph.tocsr(), norm="l1")
-> 2147 inds = csr_graph.indices.reshape(X.shape[0], self._n_neighbors)
2148 weights = csr_graph.data.reshape(X.shape[0], self._n_neighbors)
2149 embedding = init_transform(inds, weights, self.embedding_)
ValueError: cannot reshape array of size 1 into shape (1,15)
```
The only unusual thing I've found about the "bad" rows is that they have a relatively small number of non-zero entries. For example, I looked for all rows with <10 non-zero entries, and most (but not all!) of them produced the error above. If I take a random sample of rows, rather than hunting for bad ones, the failure rate is <1%.
If I can provide anything else that would be useful, please let me know!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in umap/umap_.py at transform, especially the graph normalization and reshape around line 2147. Reproduce the failure with a sparse row having few non-zero entries, then trace why the graph does not contain the expected number of neighbors. Done means transform handles these rows without the reported reshape error, with regression coverage added if the project’s test location is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100