Invalid array access in transform()
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
When using the `fit()` then `transform()` workflow, umap shows undefined behavior such as intermittently returning a matrix of all `NaN` values, segfaulting, or even returning a correct value.
Looking into the issue, it appears as though the `transform()` function shows the undefined behavior when a training data set is used to `fit()` the model and a test set (with a cardinality greater than the training set) is passed to the `transform()` function.
The culprit appears to be the modulus `n_vertices` in the `optimize_layout()` function. When the size of `tail_embedding` is less than `n_vertices`, the accelerated code may perform an invalid array access (leading to segfaults, occassionally the correct value, etc.).
I believe the problem can be fixed by passing to `optimize_layout()` the shape of the original embedding in the `transform()` function instead of the shape of the new transformation data.
I am currently unsure if this would still be valid for the algorithm (though it still seems like a valid use case). I would appreciate any feedback on this discovery, thanks!
Contributor guide
Research direction
Start by tracing transform() into optimize_layout(), comparing the original embedding shape with the test transformation data shape. Reproduce the fit() then transform() workflow where the test set is larger than the training set, and verify that array accesses remain valid without NaN output or segfaults while preserving correct results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100