`sc.external.pp.scrublet` Key error due to internal sc.pp.filter_cells/genes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 779
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 27
Description
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on the latest version of scanpy.
- (optional) I have confirmed this bug exists on the master branch of scanpy.
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
_run_scrublet sometimes removes cells, due to these lines. As a result, this line results in a key error since the scrubbed data contains fewer cells than the original adata.
Minimal code sample (that we can copy&paste without having any data)
I do not have a code sample that does not require data, but any anndata that contains cells with fewer than 3 genes should trigger the error.
import scanpy
adata = scanpy.read("test.h5ad") # e.g. contains cells with fewer than 3 genes
sc.external.pp.scrublet(test_adata, batch_key = "label")
~/.local/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:251: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
adata.var['n_cells'] = number
~/.local/lib/python3.9/site-packages/scanpy/preprocessing/_normalization.py:170: UserWarning: Received a view of an AnnData. Making a copy.
view_to_actual(adata)
Automatically set threshold at doublet score = 0.16
Detected doublet rate = 6.4%
Estimated detectable doublet fraction = 61.7%
Overall doublet rate:
Expected = 5.0%
Estimated = 10.4%
~/.local/lib/python3.9/site-packages/scanpy/preprocessing/_simple.py:251: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
adata.var['n_cells'] = number
~/.local/lib/python3.9/site-packages/scanpy/preprocessing/_normalization.py:170: UserWarning: Received a view of an AnnData. Making a copy.
view_to_actual(adata)
Automatically set threshold at doublet score = 0.17
Detected doublet rate = 5.8%
Estimated detectable doublet fraction = 55.7%
Overall doublet rate:
Expected = 5.0%
Estimated = 10.5%
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/tmp/ipykernel_2434661/1304834185.py in <module>
----> 1 sc.external.pp.scrublet(test_adata, batch_key = "label")
~/.local/lib/python3.9/site-packages/scanpy/external/pp/_scrublet.py in scrublet(adata, adata_sim, batch_key, sim_doublet_ratio, expected_doublet_rate, stdev_doublet_rate, synthetic_doublet_umi_subsampling, knn_dist_metric, normalize_variance, log_transform, mean_center, n_prin_comps, use_approx_neighbors, get_doublet_neighbor_parents, n_neighbors, threshold, verbose, copy, random_state)
250 # Now reset the obs to get the scrublet scores
251
--> 252 adata.obs = scrubbed_obs.loc[adata.obs_names.values]
253
254 # Save the .uns from each batch separately
<path>/lib/python3.9/site-packages/pandas/core/indexing.py in __getitem__(self, key)
929
930 maybe_callable = com.apply_if_callable(key, self.obj)
--> 931 return self._getitem_axis(maybe_callable, axis=axis)
932
933 def _is_scalar_access(self, key: tuple):
<path>/lib/python3.9/site-packages/pandas/core/indexing.py in _getitem_axis(self, key, axis)
1151 raise ValueError("Cannot index with multidimensional key")
1152
-> 1153 return self._getitem_iterable(key, axis=axis)
1154
1155 # nested tuple slicing
<path>/lib/python3.9/site-packages/pandas/core/indexing.py in _getitem_iterable(self, key, axis)
1091
1092 # A collection of keys
-> 1093 keyarr, indexer = self._get_listlike_indexer(key, axis)
1094 return self.obj._reindex_with_indexers(
1095 {axis: [keyarr, indexer]}, copy=True, allow_dups=True
<path>/lib/python3.9/site-packages/pandas/core/indexing.py in _get_listlike_indexer(self, key, axis)
1312 keyarr, indexer, new_indexer = ax._reindex_non_unique(keyarr)
1313
-> 1314 self._validate_read_indexer(keyarr, indexer, axis)
1315
1316 if needs_i8_conversion(ax.dtype) or isinstance(
<path>/lib/python3.9/site-packages/pandas/core/indexing.py in _validate_read_indexer(self, key, indexer, axis)
1375
1376 not_found = list(ensure_index(key)[missing_mask.nonzero()[0]].unique())
-> 1377 raise KeyError(f"{not_found} not in index")
1378
1379
KeyError: "['GGAACCCTCTCCCAGC-batch1'] not in index"
Versions
anndata 0.8.0
scanpy 1.9.1
PIL 8.4.0
annoy NA
anyio NA
attr 21.2.0
autoreload NA
babel 2.9.1
backcall 0.2.0
beta_ufunc NA
binom_ufunc NA
bottleneck 1.3.2
brotli NA
certifi 2021.10.08
cffi 1.14.6
chardet 4.0.0
charset_normalizer 2.0.4
cloudpickle 2.0.0
colorama 0.4.4
cycler 0.10.0
cython_runtime NA
cytoolz 0.11.0
dask 2021.10.0
dateutil 2.8.2
debugpy 1.4.1
decorator 5.1.0
defusedxml 0.7.1
entrypoints 0.3
fastjsonschema NA
fsspec 2021.08.1
h5py 3.3.0
idna 3.2
igraph 0.10.2
ipykernel 6.4.1
ipython_genutils 0.2.0
ipywidgets 7.6.5
jedi 0.18.0
jinja2 3.1.2
joblib 1.1.0
json5 NA
jsonschema 3.2.0
jupyter_server 1.23.3
jupyterlab_server 2.8.2
kiwisolver 1.3.1
leidenalg 0.9.0
llvmlite 0.37.0
markupsafe 2.1.1
matplotlib 3.4.3
matplotlib_inline NA
mkl 2.4.0
mpl_toolkits NA
natsort 8.2.0
nbclassic 0.4.8
nbformat 5.7.0
nbinom_ufunc NA
notebook_shim NA
numba 0.54.1
numexpr 2.7.3
numpy 1.20.3
packaging 21.0
pandas 1.3.4
parso 0.8.2
pexpect 4.8.0
pickleshare 0.7.5
pkg_resources NA
plotly 5.11.0
prometheus_client NA
prompt_toolkit 3.0.20
psutil 5.8.0
ptyprocess 0.7.0
pvectorc NA
pycparser 2.20
pydev_ipython NA
pydevconsole NA
pydevd 2.4.1
pydevd_concurrency_analyser NA
pydevd_file_utils NA
pydevd_plugins NA
pydevd_tracing NA
pyexpat NA
pygments 2.10.0
pynndescent 0.5.8
pyparsing 3.0.4
pyrsistent NA
pytz 2021.3
pywt 1.1.1
requests 2.26.0
scipy 1.7.1
scrublet NA
seaborn 0.11.2
send2trash NA
session_info 1.0.0
settings NA
simplejson 3.17.6
six 1.16.0
skimage 0.18.3
sklearn 0.24.2
sniffio 1.2.0
socks 1.7.1
sphinxcontrib NA
statsmodels 0.12.2
storemagic NA
tblib 1.7.0
terminado 0.9.4
texttable 1.6.4
tlz 0.11.0
toolz 0.11.1
tornado 6.1
tqdm 4.62.3
traitlets 5.1.0
typing_extensions NA
umap 0.5.3
urllib3 1.26.7
wcwidth 0.2.5
websocket 1.4.2
yaml 6.0
zmq 22.2.1
zope NA
IPython 7.29.0
jupyter_client 6.1.12
jupyter_core 4.8.1
jupyterlab 3.2.1
notebook 6.4.5
Python 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0]
Linux-4.18.0-305.45.1.el8_4.x86_64-x86_64-with-glibc2.28
Session information updated at 2022-12-04
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in scanpy/external/pp/_scrublet.py, especially lines 175-176 where filtering occurs and line 252 where scrubbed_obs is indexed. Reproduce with an AnnData object containing cells with fewer than 3 genes and batch_key="label"; done means sc.external.pp.scrublet completes without a KeyError and preserves consistent observation data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- bioinformatics
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100