lmcinnes / lmcinnes/umap

numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend) non-precise type pyobject

Open
#392 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

Hi,
I am having troubles in using the scanpy.tl.neighbors function on data with more than 2^13=8191 rows while using the "backed" mode.

My machine has ubuntu 18.4, I run the code with a conda environment.
I am using these versions of libraries:
scanpy 1.4.5
hdf5 1.10.4
numpy 1.17.4
anndata 0.7rc1

This is an example of code reproducing the problem:

import scanpy as sc
import numpy as np
import anndata
import pandas as pd

data=np.ones((8192,2))

obs = pd.DataFrame()
obs['obs'] = np.arange(data.shape[0])
var = pd.DataFrame()
var['var'] = np.arange(data.shape[1])
adata = anndata.AnnData(X = data, obs=obs)
adata.filename='back.h5ad'

sc.pp.neighbors(adata, n_neighbors = 10) 
print('done with knn graph')

The returned error is the following:

Traceback (most recent call last):
  File "create_files.py", line 15, in <module>
    sc.pp.neighbors(adata, n_neighbors = 10) 
  File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/scanpy/neighbors/__init__.py", line 110, in neighbors
    random_state=random_state,
  File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/scanpy/neighbors/__init__.py", line 691, in compute_neighbors
    X, n_neighbors, random_state, metric=metric, metric_kwds=metric_kwds)
  File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/scanpy/neighbors/__init__.py", line 248, in compute_neighbors_umap
    angular=angular, verbose=verbose,
  File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/umap_.py", line 269, in nearest_neighbors
    rp_forest = make_forest(X, n_neighbors, n_trees, rng_state, angular)
  File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py", line 715, in make_forest
    for i in range(n_trees)
  File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py", line 715, in <listcomp>
    for i in range(n_trees)
  File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py", line 573, in make_tree
    return make_euclidean_tree(data, indices, rng_state, leaf_size)
  File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py", line 453, in make_euclidean_tree
    data, indices, rng_state
  File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/numba/dispatcher.py", line 401, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/numba/dispatcher.py", line 344, in error_rewrite
    reraise(type(e), e, None)
  File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/numba/six.py", line 668, in reraise
    raise value.with_traceback(tb)
numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
non-precise type pyobject
[1] During: typing of argument at /home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py (160)

File "../../../../../anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py", line 160:
def euclidean_random_projection_split(data, indices, rng_state):
    <source elided>
    """
    dim = data.shape[1]
    ^

This error may have been caused by the following argument(s):
- argument 0: cannot determine Numba type of <class 'h5py._hl.dataset.Dataset'>

Additionally, I've noticed that disabling numba jit (setting the environment varable DISABLE_JIT: 0 with a yaml file named .numba_config.yaml) it works, instead.

Could you help me understand how to solve the problem?
I am available for further details if necessary.
Thanks

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the nearest_neighbors call in umap/rp_tree.py shown in the traceback and reproduce the failure using the provided 8192-row backed AnnData example. Investigate the interaction between the JIT path and the h5py Dataset, then verify that the example completes with JIT enabled without regressing ordinary array inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.