lmcinnes / lmcinnes/umap

`transform` after `fit_transform` on large dataset

Open
#65 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

Hi there,

I've been experimenting with UMAP as an alternative to t-SNE in some neuroscience applications, and I have been very happy with the combination of mathematical rigor and dedication to software engineering in this project. It's a combination very rarely seen. Keep up the fantastic work!

I'm getting a not-very-descriptive numba error in the `transform()` function in the `0.3dev` branch after fitting the model on a large dataset. See error message on [this separate gist](https://gist.github.com/bccho/0180124f555715b43eb7b64ecc3a28f6) (too long).

I'm not experienced with numba so I'm finding it hard to interpret the traceback, but from the MWE below, I think the error has something to do with the fact pairwise distances are computed [only for small datasets](https://github.com/lmcinnes/umap/blob/0.3dev/umap/umap_.py#L1204-L1220).

MWE:
```python
import numpy as np
from umap import UMAP
data = np.random.randn(10000, 10)

# doesn't work
umap_model = UMAP(verbose=True)
embedded = umap_model.fit_transform(data[0:4096, :])
reembedded = umap_model.transform(data[-100:, :]) # throws error

# does work
umap_model = UMAP(verbose=True)
embedded = umap_model.fit_transform(data[0:4095, :])
reembedded = umap_model.transform(data[-100:, :])
```

Would you be able to take a look at this, please?

Contributor guide

Open the contributing guide

Research direction

Start with the MWE and the linked section of umap/umap_.py at lines 1204-1220, then compare the 4096-row and 4095-row cases with the full traceback in the gist. The issue is done when transform() works after fitting on the 4096-row dataset and the large-dataset error is covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.