lmcinnes / lmcinnes/umap

Invalid array access in transform()

Open
#89 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

When using the `fit()` then `transform()` workflow, umap shows undefined behavior such as intermittently returning a matrix of all `NaN` values, segfaulting, or even returning a correct value.

Looking into the issue, it appears as though the `transform()` function shows the undefined behavior when a training data set is used to `fit()` the model and a test set (with a cardinality greater than the training set) is passed to the `transform()` function.

The culprit appears to be the modulus `n_vertices` in the `optimize_layout()` function. When the size of `tail_embedding` is less than `n_vertices`, the accelerated code may perform an invalid array access (leading to segfaults, occassionally the correct value, etc.).

I believe the problem can be fixed by passing to `optimize_layout()` the shape of the original embedding in the `transform()` function instead of the shape of the new transformation data.

I am currently unsure if this would still be valid for the algorithm (though it still seems like a valid use case). I would appreciate any feedback on this discovery, thanks!

Contributor guide

Open the contributing guide

Research direction

Start by tracing transform() into optimize_layout(), comparing the original embedding shape with the test transformation data shape. Reproduce the fit() then transform() workflow where the test set is larger than the training set, and verify that array accesses remain valid without NaN output or segfaults while preserving correct results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.