`transform` after `fit_transform` on large dataset
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
Hi there,
I've been experimenting with UMAP as an alternative to t-SNE in some neuroscience applications, and I have been very happy with the combination of mathematical rigor and dedication to software engineering in this project. It's a combination very rarely seen. Keep up the fantastic work!
I'm getting a not-very-descriptive numba error in the `transform()` function in the `0.3dev` branch after fitting the model on a large dataset. See error message on [this separate gist](https://gist.github.com/bccho/0180124f555715b43eb7b64ecc3a28f6) (too long).
I'm not experienced with numba so I'm finding it hard to interpret the traceback, but from the MWE below, I think the error has something to do with the fact pairwise distances are computed [only for small datasets](https://github.com/lmcinnes/umap/blob/0.3dev/umap/umap_.py#L1204-L1220).
MWE:
```python
import numpy as np
from umap import UMAP
data = np.random.randn(10000, 10)
# doesn't work
umap_model = UMAP(verbose=True)
embedded = umap_model.fit_transform(data[0:4096, :])
reembedded = umap_model.transform(data[-100:, :]) # throws error
# does work
umap_model = UMAP(verbose=True)
embedded = umap_model.fit_transform(data[0:4095, :])
reembedded = umap_model.transform(data[-100:, :])
```
Would you be able to take a look at this, please?
Contributor guide
Research direction
Start with the MWE and the linked section of umap/umap_.py at lines 1204-1220, then compare the 4096-row and 4095-row cases with the full traceback in the gist. The issue is done when transform() works after fitting on the 4096-row dataset and the large-dataset error is covered by an appropriate regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100