Tools dendogram fails when using categorical obs key where not all categories have values
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 779
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 27
Description
In certain scenarios sc.tl.dendrogram is used with an obs key which is categorial, but not all categories are used in the specific slicing of use.
Because the current code doesn't check if all the categories are used by the current samples selection, it creates a non-symmetric matrix when line 133 of _dendogram.py creates mean_df = rep_df.groupby(level=0).mean()
maybe some solution could be found using remove_unused_categories
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on the latest version of scanpy.
- (optional) I have confirmed this bug exists on the master branch of scanpy.
Minimal code sample (that we can copy&paste without having any data)
import pandas as pd
import numpy as np
import scanpy as sc
obs = pd.DataFrame(pd.Categorical(["a","b"], categories=["a","b","c"]), columns=['cat_key'])
adata = sc.AnnData(np.ones((2, 1)), obs=obs)
sc.tl.dendrogram(adata, 'cat_key', use_rep='X')
Traceback (most recent call last):
File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3442, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-8-1616c771d0da>", line 1, in <module>
sc.tl.dendrogram(adata, 'cat_key', use_rep='X')
File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/scanpy/tools/_dendrogram.py", line 139, in dendrogram
corr_condensed = distance.squareform(1 - corr_matrix)
File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/scipy/spatial/distance.py", line 2354, in squareform
is_valid_dm(X, throw=True, name='X')
File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/scipy/spatial/distance.py", line 2429, in is_valid_dm
raise ValueError(('Distance matrix \'%s\' must be '
ValueError: Distance matrix 'X' must be symmetric.```
#### Versions
<details>
sc.logging.print_versions()
-----
anndata 0.8.0
scanpy 1.9.1
-----
PIL 9.4.0
PyQt5 NA
anyio NA
asttokens NA
attr 22.2.0
babel 2.11.0
backcall 0.2.0
beta_ufunc NA
binom_ufunc NA
bottleneck 1.3.6
brotli NA
certifi 2022.12.07
cffi 1.15.1
charset_normalizer 2.1.1
cloudpickle 2.2.1
colorama 0.4.6
comm 0.1.2
cycler 0.10.0
cython_runtime NA
cytoolz 0.12.0
dask 2023.1.1
dateutil 2.8.2
decorator 5.1.1
defusedxml 0.7.1
executing 1.2.0
fastcluster 1.2.6
fastjsonschema NA
gepdynamics NA
h5py 3.8.0
hypergeom_ufunc NA
idna 3.4
igraph 0.10.4
invgauss_ufunc NA
ipykernel 6.21.1
ipython_genutils 0.2.0
ipywidgets 8.0.4
jedi 0.18.2
jinja2 3.1.2
joblib 1.2.0
json5 NA
jsonschema 4.17.3
jupyter_events 0.5.0
jupyter_server 2.2.1
jupyterlab_server 2.19.0
kiwisolver 1.4.4
leidenalg 0.9.1
llvmlite 0.39.1
louvain 0.8.0
markupsafe 2.1.2
matplotlib 3.6.3
mpl_toolkits NA
natsort 8.2.0
nbformat 5.7.3
nbinom_ufunc NA
ncf_ufunc NA
nct_ufunc NA
ncx2_ufunc NA
numba 0.56.4
numexpr 2.8.3
numpy 1.23.5
packaging 23.0
pandas 1.5.3
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
pkg_resources NA
platformdirs 2.6.2
prometheus_client NA
prompt_toolkit 3.0.36
psutil 5.9.4
ptyprocess 0.7.0
pure_eval 0.2.2
pvectorc NA
pydev_ipython NA
pydevd 1.4.0
pydevd_concurrency_analyser NA
pydevd_file_utils NA
pydevd_plugins NA
pydevd_tracing NA
pygments 2.14.0
pyparsing 3.0.9
pyrsistent NA
pythonjsonlogger NA
pytz 2022.7.1
requests 2.28.2
scipy 1.10.0
seaborn 0.12.2
send2trash NA
session_info 1.0.0
setuptools 67.1.0
sip NA
sitecustomize NA
six 1.16.0
skewnorm_ufunc NA
sklearn 1.2.1
sniffio 1.3.0
socks 1.7.1
stack_data 0.6.2
statsmodels 0.13.5
texttable 1.6.7
threadpoolctl 3.1.0
tlz 0.12.0
toolz 0.12.0
tornado 6.2
traitlets 5.9.0
typing_extensions NA
unicodedata2 NA
urllib3 1.26.14
wcwidth 0.2.6
websocket 1.5.1
yaml 6.0
zipp NA
zmq 25.0.0
zoneinfo NA
-----
IPython 8.9.0
jupyter_client 8.0.2
jupyter_core 5.2.0
jupyterlab 3.6.1
notebook 6.5.2
-----
Python 3.9.16 | packaged by conda-forge | (main, Feb 1 2023, 21:42:20) [Clang 14.0.6 ]
macOS-13.2.1-x86_64-i386-64bit
-----
Session information updated at 2023-02-15 22:51
</details>
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in scanpy/tools/_dendrogram.py around line 133, where the grouped means are created, and reproduce the failure with the minimal categorical obs example in the issue. The fix is complete when sc.tl.dendrogram runs for categories with no values in the selection and the resulting distance matrix is symmetric.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-visualization
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100