numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend) non-precise type pyobject
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
Hi,
I am having troubles in using the scanpy.tl.neighbors function on data with more than 2^13=8191 rows while using the "backed" mode.
My machine has ubuntu 18.4, I run the code with a conda environment.
I am using these versions of libraries:
scanpy 1.4.5
hdf5 1.10.4
numpy 1.17.4
anndata 0.7rc1
This is an example of code reproducing the problem:
import scanpy as sc
import numpy as np
import anndata
import pandas as pd
data=np.ones((8192,2))
obs = pd.DataFrame()
obs['obs'] = np.arange(data.shape[0])
var = pd.DataFrame()
var['var'] = np.arange(data.shape[1])
adata = anndata.AnnData(X = data, obs=obs)
adata.filename='back.h5ad'
sc.pp.neighbors(adata, n_neighbors = 10)
print('done with knn graph')
The returned error is the following:
Traceback (most recent call last):
File "create_files.py", line 15, in <module>
sc.pp.neighbors(adata, n_neighbors = 10)
File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/scanpy/neighbors/__init__.py", line 110, in neighbors
random_state=random_state,
File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/scanpy/neighbors/__init__.py", line 691, in compute_neighbors
X, n_neighbors, random_state, metric=metric, metric_kwds=metric_kwds)
File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/scanpy/neighbors/__init__.py", line 248, in compute_neighbors_umap
angular=angular, verbose=verbose,
File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/umap_.py", line 269, in nearest_neighbors
rp_forest = make_forest(X, n_neighbors, n_trees, rng_state, angular)
File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py", line 715, in make_forest
for i in range(n_trees)
File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py", line 715, in <listcomp>
for i in range(n_trees)
File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py", line 573, in make_tree
return make_euclidean_tree(data, indices, rng_state, leaf_size)
File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py", line 453, in make_euclidean_tree
data, indices, rng_state
File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/numba/dispatcher.py", line 401, in _compile_for_args
error_rewrite(e, 'typing')
File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/numba/dispatcher.py", line 344, in error_rewrite
reraise(type(e), e, None)
File "/home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/numba/six.py", line 668, in reraise
raise value.with_traceback(tb)
numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
non-precise type pyobject
[1] During: typing of argument at /home/sara_f/anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py (160)
File "../../../../../anaconda3/envs/sci/lib/python3.7/site-packages/umap/rp_tree.py", line 160:
def euclidean_random_projection_split(data, indices, rng_state):
<source elided>
"""
dim = data.shape[1]
^
This error may have been caused by the following argument(s):
- argument 0: cannot determine Numba type of <class 'h5py._hl.dataset.Dataset'>
Additionally, I've noticed that disabling numba jit (setting the environment varable DISABLE_JIT: 0 with a yaml file named .numba_config.yaml) it works, instead.
Could you help me understand how to solve the problem?
I am available for further details if necessary.
Thanks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the nearest_neighbors call in umap/rp_tree.py shown in the traceback and reproduce the failure using the provided 8192-row backed AnnData example. Investigate the interaction between the JIT path and the h5py Dataset, then verify that the example completes with JIT enabled without regressing ordinary array inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100