pairwise mahalanobis distance metric computation returns NaN
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
Hi.
I'm attempting to run UMAP on a specific dataset that I sample from.
For small sample sizes, it consistently works.
For larger sample sizes, it consistently fails in a manner that resembled #136.
Setup
Python: 3.9.5 (Linux/x64 on custom desktop server, Intel(R) Core(TM) i9-9900X CPU @ 3.50GHz, 125GB RAM)
UMAP: 0.5.1
Metric: "Mahalanobis"
Input: (409341, 60) dense float matrix. (The last 30 columns are 0/1 numeric values) but everything is dtype float64
When I was debugging, I noticed this:

As can be seen, the inputs are numerical values, no np.inf of np.nan values, yet the output of the distance calculation results in np.nan.
I highly suspect that the reason is that I'm actually not using any extra arguments for the Mahalanobis distance, thus the covariance matrix is the "mock_identity" which is a 2x2 matrix.
https://github.com/lmcinnes/umap/blob/47b585cd27f4aa45737f91b4f0eb4aeea6bcc3d6/umap/distances.py#L9
And judging by the code under the following for loop (and given that my X.shape[0] == 60):
https://github.com/lmcinnes/umap/blob/47b585cd27f4aa45737f91b4f0eb4aeea6bcc3d6/umap/distances.py#L282
I think it'll get an out-of-bounds error.
But because of the Numba I can't debug and just get back nan values? 😬
Unfortunately, I can't upload the data itself due to licensing so it would help to know your thoughts about it.
Also, this theory doesn't explain why it manages to work on smaller sample sizes 😞
Thanks,
Sammy.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in umap/distances.py at the Mahalanobis distance implementation and the loop around line 282, then reproduce with a dense float64 matrix having 60 features and increasing sample sizes. Check how the covariance argument and Numba execution affect the returned distances, and use finite-distance behavior on the reported input shape as the completion criterion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100