lmcinnes / lmcinnes/umap

Cannot transform sparse data with Jaccard metric

Open
#680 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

UMAP can fit sparse data with the Jaccard metric but it cannot transform new sparse data. Minimum example below:

```
from scipy.sparse import random
from umap import UMAP

# create random training data
train = random(m = 100, n = 1000, density = 0.01)

# initialize model
mapper = UMAP(metric = 'jaccard')

# fit UMAP with Jaccard metric
embedding = mapper.fit_transform(train)

# transform training data
output = mapper.transform(train)
```

Interestingly, `mapper.fit_transform` works fine but `mapper.transform` fails with the following error:

```
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/umap/umap_.py in transform(self, X)
2700 dmat = pairwise_distances(
-> 2701 X, self._raw_data, metric=_m, **self._metric_kwds
2702 )

/opt/conda/lib/python3.7/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
62 if extra_args <= 0:
---> 63 return f(*args, **kwargs)
64

/opt/conda/lib/python3.7/site-packages/sklearn/metrics/pairwise.py in pairwise_distances(X, Y, metric, n_jobs, force_all_finite, **kwds)
1767 if issparse(X) or issparse(Y):
-> 1768 raise TypeError("scipy distance metrics do not"
1769 " support sparse matrices.")

TypeError: scipy distance metrics do not support sparse matrices.

During handling of the above exception, another exception occurred:

TypingError Traceback (most recent call last)
in
----> 1 mapper.transform(train)

/opt/conda/lib/python3.7/site-packages/umap/umap_.py in transform(self, X)
2706 self._raw_data,
2707 metric=self._input_distance_func,
-> 2708 kwds=self._metric_kwds,
2709 )
2710 indices = np.argpartition(dmat, self._n_neighbors)[:, : self._n_neighbors]

/opt/conda/lib/python3.7/site-packages/umap/distances.py in pairwise_special_metric(X, Y, metric, kwds)
1260 return metric(_X, _Y, *kwd_vals)
1261
-> 1262 return pairwise_distances(X, Y, metric=_partial_metric)
1263 else:
1264 special_metric_func = named_distances[metric]

/opt/conda/lib/python3.7/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
61 extra_args = len(args) - len(all_args)
62 if extra_args <= 0:
---> 63 return f(*args, **kwargs)
64
65 # extra_args > 0

/opt/conda/lib/python3.7/site-packages/sklearn/metrics/pairwise.py in pairwise_distances(X, Y, metric, n_jobs, force_all_finite, **kwds)
1788 func = partial(distance.cdist, metric=metric, **kwds)
1789
-> 1790 return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
1791
1792

/opt/conda/lib/python3.7/site-packages/sklearn/metrics/pairwise.py in _parallel_pairwise(X, Y, func, n_jobs, **kwds)
1357
1358 if effective_n_jobs(n_jobs) == 1:
-> 1359 return func(X, Y, **kwds)
1360
1361 # enforce a threading backend to prevent data communication overhead

/opt/conda/lib/python3.7/site-packages/sklearn/metrics/pairwise.py in _pairwise_callable(X, Y, metric, force_all_finite, **kwds)
1401 iterator = itertools.product(range(X.shape[0]), range(Y.shape[0]))
1402 for i, j in iterator:
-> 1403 out[i, j] = metric(X[i], Y[j], **kwds)
1404
1405 return out

/opt/conda/lib/python3.7/site-packages/numba/core/dispatcher.py in _compile_for_args(self, *args, **kws)
412 e.patch_message(msg)
413
--> 414 error_rewrite(e, 'typing')
415 except errors.UnsupportedError as e:
416 # Something unsupported is present in the user code, add help info

/opt/conda/lib/python3.7/site-packages/numba/core/dispatcher.py in error_rewrite(e, issue_type)
355 raise e
356 else:
--> 357 raise e.with_traceback(None)
358
359 argtypes = []

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
non-precise type pyobject
During: typing of argument at /opt/conda/lib/python3.7/site-packages/umap/distances.py (1260)

File "../../opt/conda/lib/python3.7/site-packages/umap/distances.py", line 1260:
def _partial_metric(_X, _Y=None):
return metric(_X, _Y, *kwd_vals)
^

This error may have been caused by the following argument(s):
- argument 0: Cannot determine Numba type of
- argument 1: Cannot determine Numba type of
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with transform in umap/umap_.py and follow its sparse-distance fallback into pairwise_special_metric in umap/distances.py. Reproduce the supplied sparse Jaccard example and verify that mapper.transform(train) completes without the reported TypeError or Numba TypingError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.