scverse / scverse/scanpy

`sc.external.pp.scrublet` Key error due to internal sc.pp.filter_cells/genes

Open
#2,377 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.6k
Forks
779
Avg merge
1d 4h
Merged PRs (30d)
27

Description

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of scanpy.
  • (optional) I have confirmed this bug exists on the master branch of scanpy.

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

_run_scrublet sometimes removes cells, due to these lines. As a result, this line results in a key error since the scrubbed data contains fewer cells than the original adata.

Minimal code sample (that we can copy&paste without having any data)

I do not have a code sample that does not require data, but any anndata that contains cells with fewer than 3 genes should trigger the error.

import scanpy
adata = scanpy.read("test.h5ad") # e.g. contains cells with fewer than 3 genes
sc.external.pp.scrublet(test_adata, batch_key = "label")
~/.local/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:251: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var['n_cells'] = number
~/.local/lib/python3.9/site-packages/scanpy/preprocessing/_normalization.py:170: UserWarning: Received a view of an AnnData. Making a copy.
  view_to_actual(adata)

Automatically set threshold at doublet score = 0.16
Detected doublet rate = 6.4%
Estimated detectable doublet fraction = 61.7%
Overall doublet rate:
	Expected   = 5.0%
	Estimated  = 10.4%

~/.local/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:251: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var['n_cells'] = number
~/.local/lib/python3.9/site-packages/scanpy/preprocessing/_normalization.py:170: UserWarning: Received a view of an AnnData. Making a copy.
  view_to_actual(adata)

Automatically set threshold at doublet score = 0.17
Detected doublet rate = 5.8%
Estimated detectable doublet fraction = 55.7%
Overall doublet rate:
	Expected   = 5.0%
	Estimated  = 10.5%

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/tmp/ipykernel_2434661/1304834185.py in <module>
----> 1 sc.external.pp.scrublet(test_adata, batch_key = "label")

~/.local/lib/python3.9/site-packages/scanpy/external/pp/_scrublet.py in scrublet(adata, adata_sim, batch_key, sim_doublet_ratio, expected_doublet_rate, stdev_doublet_rate, synthetic_doublet_umi_subsampling, knn_dist_metric, normalize_variance, log_transform, mean_center, n_prin_comps, use_approx_neighbors, get_doublet_neighbor_parents, n_neighbors, threshold, verbose, copy, random_state)
    250         # Now reset the obs to get the scrublet scores
    251 
--> 252         adata.obs = scrubbed_obs.loc[adata.obs_names.values]
    253 
    254         # Save the .uns from each batch separately

<path>/lib/python3.9/site-packages/pandas/core/indexing.py in __getitem__(self, key)
    929 
    930             maybe_callable = com.apply_if_callable(key, self.obj)
--> 931             return self._getitem_axis(maybe_callable, axis=axis)
    932 
    933     def _is_scalar_access(self, key: tuple):

<path>/lib/python3.9/site-packages/pandas/core/indexing.py in _getitem_axis(self, key, axis)
   1151                     raise ValueError("Cannot index with multidimensional key")
   1152 
-> 1153                 return self._getitem_iterable(key, axis=axis)
   1154 
   1155             # nested tuple slicing

<path>/lib/python3.9/site-packages/pandas/core/indexing.py in _getitem_iterable(self, key, axis)
   1091 
   1092         # A collection of keys
-> 1093         keyarr, indexer = self._get_listlike_indexer(key, axis)
   1094         return self.obj._reindex_with_indexers(
   1095             {axis: [keyarr, indexer]}, copy=True, allow_dups=True

<path>/lib/python3.9/site-packages/pandas/core/indexing.py in _get_listlike_indexer(self, key, axis)
   1312             keyarr, indexer, new_indexer = ax._reindex_non_unique(keyarr)
   1313 
-> 1314         self._validate_read_indexer(keyarr, indexer, axis)
   1315 
   1316         if needs_i8_conversion(ax.dtype) or isinstance(

<path>/lib/python3.9/site-packages/pandas/core/indexing.py in _validate_read_indexer(self, key, indexer, axis)
   1375 
   1376             not_found = list(ensure_index(key)[missing_mask.nonzero()[0]].unique())
-> 1377             raise KeyError(f"{not_found} not in index")
   1378 
   1379 

KeyError: "['GGAACCCTCTCCCAGC-batch1'] not in index"
Versions

anndata 0.8.0
scanpy 1.9.1

PIL 8.4.0
annoy NA
anyio NA
attr 21.2.0
autoreload NA
babel 2.9.1
backcall 0.2.0
beta_ufunc NA
binom_ufunc NA
bottleneck 1.3.2
brotli NA
certifi 2021.10.08
cffi 1.14.6
chardet 4.0.0
charset_normalizer 2.0.4
cloudpickle 2.0.0
colorama 0.4.4
cycler 0.10.0
cython_runtime NA
cytoolz 0.11.0
dask 2021.10.0
dateutil 2.8.2
debugpy 1.4.1
decorator 5.1.0
defusedxml 0.7.1
entrypoints 0.3
fastjsonschema NA
fsspec 2021.08.1
h5py 3.3.0
idna 3.2
igraph 0.10.2
ipykernel 6.4.1
ipython_genutils 0.2.0
ipywidgets 7.6.5
jedi 0.18.0
jinja2 3.1.2
joblib 1.1.0
json5 NA
jsonschema 3.2.0
jupyter_server 1.23.3
jupyterlab_server 2.8.2
kiwisolver 1.3.1
leidenalg 0.9.0
llvmlite 0.37.0
markupsafe 2.1.1
matplotlib 3.4.3
matplotlib_inline NA
mkl 2.4.0
mpl_toolkits NA
natsort 8.2.0
nbclassic 0.4.8
nbformat 5.7.0
nbinom_ufunc NA
notebook_shim NA
numba 0.54.1
numexpr 2.7.3
numpy 1.20.3
packaging 21.0
pandas 1.3.4
parso 0.8.2
pexpect 4.8.0
pickleshare 0.7.5
pkg_resources NA
plotly 5.11.0
prometheus_client NA
prompt_toolkit 3.0.20
psutil 5.8.0
ptyprocess 0.7.0
pvectorc NA
pycparser 2.20
pydev_ipython NA
pydevconsole NA
pydevd 2.4.1
pydevd_concurrency_analyser NA
pydevd_file_utils NA
pydevd_plugins NA
pydevd_tracing NA
pyexpat NA
pygments 2.10.0
pynndescent 0.5.8
pyparsing 3.0.4
pyrsistent NA
pytz 2021.3
pywt 1.1.1
requests 2.26.0
scipy 1.7.1
scrublet NA
seaborn 0.11.2
send2trash NA
session_info 1.0.0
settings NA
simplejson 3.17.6
six 1.16.0
skimage 0.18.3
sklearn 0.24.2
sniffio 1.2.0
socks 1.7.1
sphinxcontrib NA
statsmodels 0.12.2
storemagic NA
tblib 1.7.0
terminado 0.9.4
texttable 1.6.4
tlz 0.11.0
toolz 0.11.1
tornado 6.1
tqdm 4.62.3
traitlets 5.1.0
typing_extensions NA
umap 0.5.3
urllib3 1.26.7
wcwidth 0.2.5
websocket 1.4.2
yaml 6.0
zmq 22.2.1
zope NA

IPython 7.29.0
jupyter_client 6.1.12
jupyter_core 4.8.1
jupyterlab 3.2.1
notebook 6.4.5

Python 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0]
Linux-4.18.0-305.45.1.el8_4.x86_64-x86_64-with-glibc2.28

Session information updated at 2022-12-04

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in scanpy/external/pp/_scrublet.py, especially lines 175-176 where filtering occurs and line 252 where scrubbed_obs is indexed. Reproduce with an AnnData object containing cells with fewer than 3 genes and batch_key="label"; done means sc.external.pp.scrublet completes without a KeyError and preserves consistent observation data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
bioinformatics
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.