lmcinnes / lmcinnes/umap

“ValueError: cannot reshape array of size ...” from transform

Open
#442 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

First, thanks for putting such a great project together!

I'm having trouble with the transform method. I'm training on TFIDF vectors from a large set of text data (>100k rows, 20k features), of type scipy.sparse.csr.csr_matrix. I'm then applying to new set of test data encoded in the same way. I seem to have some "bad" rows, and get an error if I try to transform the full test set, or an individual "bad" row. That error is:

```
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
in
----> 1 embedding_n.transform(tfidf_test[30103])

/projects/PK13588/CallReason_Exploratory/envs/ffs/lib/python3.6/site-packages/umap/umap_.py in transform(self, X)
2145 # and data. Doing so relies on the constant degree assumption!
2146 csr_graph = normalize(graph.tocsr(), norm="l1")
-> 2147 inds = csr_graph.indices.reshape(X.shape[0], self._n_neighbors)
2148 weights = csr_graph.data.reshape(X.shape[0], self._n_neighbors)
2149 embedding = init_transform(inds, weights, self.embedding_)

ValueError: cannot reshape array of size 1 into shape (1,15)
```
The only unusual thing I've found about the "bad" rows is that they have a relatively small number of non-zero entries. For example, I looked for all rows with <10 non-zero entries, and most (but not all!) of them produced the error above. If I take a random sample of rows, rather than hunting for bad ones, the failure rate is <1%.

If I can provide anything else that would be useful, please let me know!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in umap/umap_.py at transform, especially the graph normalization and reshape around line 2147. Reproduce the failure with a sparse row having few non-zero entries, then trace why the graph does not contain the expected number of neighbors. Done means transform handles these rows without the reported reshape error, with regression coverage added if the project’s test location is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.