lmcinnes / lmcinnes/umap

UMAP segmentation faults with correlation and cosine metrics in certain datasets

Open
#956 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

UMAP version 0.5.3
numba version 0.56.4

UMAP segfaults with correlation and cosine metrics but not with other metrics (euclidean, manhattan and camberra tested). The issues only occurs with very specific datasets.

Here is my code:

clusterable_embedding_test = umap.UMAP(
    n_neighbors=30,
    min_dist=.0,
    n_components=9,
    random_state=31416,
    #metric='manhattan',
    #metric='canberra',
    #metric='cosine',
    metric='correlation',
).fit_transform(df_clean)

I have tested in native Ubuntu 22.04, Ubuntu 22.04 under WSL2 and Windows. Same issue.

The issue only manifests itself when I remove a specific column from the dataset. If the column is present the code runs fine.

It does not matter how the column is removed (whether dropped from the dataframe, not loaded or removed beforehand from the CSV file).

Removing or adding other columns is OK, regardless of how many.

Setting NUMBA_DISABLE_JIT=1 prevents the segfault but results in extremely slow execution.

NUMBA_DISABLE_INTEL_SVML=1 has no effect. Same for a large number of other numba-related variables I have tried.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with UMAP 0.5.3 and numba 0.56.4 using the shown UMAP configuration, comparing correlation and cosine with the other tested metrics. Investigate why removing the specific dataset column triggers a native segfault and verify behavior across the reported environments. Done means the failing dataset no longer segfaults without disabling JIT, with a regression test or reproducible case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.