scverse / scverse/scanpy

Tools dendogram fails when using categorical obs key where not all categories have values

Open
#2,418 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.6k
Forks
779
Avg merge
1d 4h
Merged PRs (30d)
27

Description

In certain scenarios sc.tl.dendrogram is used with an obs key which is categorial, but not all categories are used in the specific slicing of use.
Because the current code doesn't check if all the categories are used by the current samples selection, it creates a non-symmetric matrix when line 133 of _dendogram.py creates mean_df = rep_df.groupby(level=0).mean()

maybe some solution could be found using remove_unused_categories

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of scanpy.
  • (optional) I have confirmed this bug exists on the master branch of scanpy.

Minimal code sample (that we can copy&paste without having any data)
import pandas as pd
import numpy as np
import scanpy as sc

obs = pd.DataFrame(pd.Categorical(["a","b"], categories=["a","b","c"]), columns=['cat_key'])
adata = sc.AnnData(np.ones((2, 1)), obs=obs)
sc.tl.dendrogram(adata, 'cat_key', use_rep='X')
Traceback (most recent call last):
  File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3442, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-8-1616c771d0da>", line 1, in <module>
    sc.tl.dendrogram(adata, 'cat_key', use_rep='X')
  File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/scanpy/tools/_dendrogram.py", line 139, in dendrogram
    corr_condensed = distance.squareform(1 - corr_matrix)
  File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/scipy/spatial/distance.py", line 2354, in squareform
    is_valid_dm(X, throw=True, name='X')
  File "/Applications/miniconda3/envs/gep-dynamics/lib/python3.9/site-packages/scipy/spatial/distance.py", line 2429, in is_valid_dm
    raise ValueError(('Distance matrix \'%s\' must be '
ValueError: Distance matrix 'X' must be symmetric.```

#### Versions

<details>

sc.logging.print_versions() 
-----
anndata     0.8.0
scanpy      1.9.1
-----
PIL                         9.4.0
PyQt5                       NA
anyio                       NA
asttokens                   NA
attr                        22.2.0
babel                       2.11.0
backcall                    0.2.0
beta_ufunc                  NA
binom_ufunc                 NA
bottleneck                  1.3.6
brotli                      NA
certifi                     2022.12.07
cffi                        1.15.1
charset_normalizer          2.1.1
cloudpickle                 2.2.1
colorama                    0.4.6
comm                        0.1.2
cycler                      0.10.0
cython_runtime              NA
cytoolz                     0.12.0
dask                        2023.1.1
dateutil                    2.8.2
decorator                   5.1.1
defusedxml                  0.7.1
executing                   1.2.0
fastcluster                 1.2.6
fastjsonschema              NA
gepdynamics                 NA
h5py                        3.8.0
hypergeom_ufunc             NA
idna                        3.4
igraph                      0.10.4
invgauss_ufunc              NA
ipykernel                   6.21.1
ipython_genutils            0.2.0
ipywidgets                  8.0.4
jedi                        0.18.2
jinja2                      3.1.2
joblib                      1.2.0
json5                       NA
jsonschema                  4.17.3
jupyter_events              0.5.0
jupyter_server              2.2.1
jupyterlab_server           2.19.0
kiwisolver                  1.4.4
leidenalg                   0.9.1
llvmlite                    0.39.1
louvain                     0.8.0
markupsafe                  2.1.2
matplotlib                  3.6.3
mpl_toolkits                NA
natsort                     8.2.0
nbformat                    5.7.3
nbinom_ufunc                NA
ncf_ufunc                   NA
nct_ufunc                   NA
ncx2_ufunc                  NA
numba                       0.56.4
numexpr                     2.8.3
numpy                       1.23.5
packaging                   23.0
pandas                      1.5.3
parso                       0.8.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
platformdirs                2.6.2
prometheus_client           NA
prompt_toolkit              3.0.36
psutil                      5.9.4
ptyprocess                  0.7.0
pure_eval                   0.2.2
pvectorc                    NA
pydev_ipython               NA
pydevd                      1.4.0
pydevd_concurrency_analyser NA
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.14.0
pyparsing                   3.0.9
pyrsistent                  NA
pythonjsonlogger            NA
pytz                        2022.7.1
requests                    2.28.2
scipy                       1.10.0
seaborn                     0.12.2
send2trash                  NA
session_info                1.0.0
setuptools                  67.1.0
sip                         NA
sitecustomize               NA
six                         1.16.0
skewnorm_ufunc              NA
sklearn                     1.2.1
sniffio                     1.3.0
socks                       1.7.1
stack_data                  0.6.2
statsmodels                 0.13.5
texttable                   1.6.7
threadpoolctl               3.1.0
tlz                         0.12.0
toolz                       0.12.0
tornado                     6.2
traitlets                   5.9.0
typing_extensions           NA
unicodedata2                NA
urllib3                     1.26.14
wcwidth                     0.2.6
websocket                   1.5.1
yaml                        6.0
zipp                        NA
zmq                         25.0.0
zoneinfo                    NA
-----
IPython             8.9.0
jupyter_client      8.0.2
jupyter_core        5.2.0
jupyterlab          3.6.1
notebook            6.5.2
-----
Python 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:42:20) [Clang 14.0.6 ]
macOS-13.2.1-x86_64-i386-64bit
-----
Session information updated at 2023-02-15 22:51
</details>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in scanpy/tools/_dendrogram.py around line 133, where the grouped means are created, and reproduce the failure with the minimal categorical obs example in the issue. The fix is complete when sc.tl.dendrogram runs for categories with no values in the selection and the resulting distance matrix is symmetric.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-visualization
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.