sc.pp.highly_variable_genes produces different hvg when run on same object with same n_genes
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 779
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 27
Description
Please make sure these conditions are met
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on the latest version of scanpy.
- (optional) I have confirmed this bug exists on the master branch of scanpy.
What happened?
I would expect that when you call sc.pp.highly_variable_genes on the same dataset and request the same number of genes, that you would get the same output. The below example suggests that this is not the case.
Minimal code sample
adata_sub = sc.read_h5ad("your_favourite_object.h5ad")
n_genes = 1491
for i in range(10):
sc.pp.highly_variable_genes(adata_sub, n_top_genes=n_genes)
unique_genes = list(adata_sub.var['highly_variable'][adata_sub.var['highly_variable'] == True].index)
if i == 0:
all_unique = list(set(unique_genes))
print(f"total {len(all_unique)} unique genes")
else:
all_unique = list(set(all_unique+unique_genes))
print(f"total {len(all_unique)} unique genes")
Error output
total 1491 unique genes
total 1814 unique genes
total 2042 unique genes
total 2163 unique genes
total 2237 unique genes
total 2305 unique genes
total 2356 unique genes
total 2401 unique genes
total 2437 unique genes
total 2453 unique genes
Versions
-----
anndata 0.9.1
scanpy 1.9.3
-----
PIL 8.4.0
asciitree NA
asttokens NA
backcall 0.2.0
beta_ufunc NA
binom_ufunc NA
bottleneck 1.3.4
cffi 1.15.0
cloudpickle 2.0.0
colorama 0.4.4
cycler 0.10.0
cython_runtime NA
cytoolz 0.11.0
dask 2022.02.1
dateutil 2.8.1
debugpy 1.5.1
decorator 5.1.1
defusedxml 0.7.1
django 4.1.3
entrypoints 0.4
executing 0.8.3
fasteners 0.18
fsspec 2023.4.0
google NA
h5py 3.6.0
igraph 0.10.6
ipykernel 6.9.1
ipython_genutils 0.2.0
ipywidgets 7.6.5
jedi 0.18.1
jinja2 3.1.2
joblib 1.1.0
jupyter_server 1.13.5
kiwisolver 1.2.0
kneed 0.8.3
leidenalg 0.10.1
llvmlite 0.38.0
markupsafe 2.0.1
matplotlib 3.5.1
matplotlib_inline NA
mishalpy NA
mpl_toolkits NA
mpmath 1.2.1
msgpack 1.0.2
natsort 8.3.1
nbinom_ufunc NA
nt NA
ntsecuritycon NA
numba 0.55.1
numcodecs 0.11.0
numexpr 2.8.1
numpy 1.21.6
opt_einsum v3.3.0
packaging 21.3
pandas 1.4.3
parso 0.8.3
pickleshare 0.7.5
pkg_resources NA
plotly 5.6.0
prompt_toolkit 3.0.20
psutil 5.8.0
pure_eval 0.2.2
pycparser 2.21
pydev_ipython NA
pydevconsole NA
pydevd 2.6.0
pydevd_concurrency_analyser NA
pydevd_file_utils NA
pydevd_plugins NA
pydevd_tracing NA
pygments 2.15.1
pyparsing 2.4.7
pythoncom NA
pytz 2020.1
pywintypes NA
ruamel NA
scipy 1.7.3
seaborn 0.11.2
session_info 1.0.0
setuptools 61.2.0
six 1.15.0
sklearn 1.0.2
snappy NA
sparse 0.14.0
sphinxcontrib NA
stack_data 0.2.0
statsmodels 0.13.2
sympy 1.10.1
tblib 1.7.0
texttable 1.6.7
threadpoolctl 2.2.0
tlz 0.11.0
toolz 0.11.2
torch 2.0.1+cpu
tornado 6.1
tqdm 4.64.0
traitlets 5.9.0
typing_extensions NA
wcwidth 0.2.5
win32api NA
win32com NA
win32security NA
yaml 6.0
zarr 2.14.2
zipp NA
zmq 22.3.0
zope NA
-----
IPython 8.2.0
jupyter_client 6.1.12
jupyter_core 4.9.2
jupyterlab 3.3.2
notebook 6.4.8
-----
Python 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)]
Windows-10-10.0.22000-SP0
-----
Session information updated at 2023-07-31 10:13
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the loop in the issue with sc.pp.highly_variable_genes on the same AnnData object and n_top_genes=1491. Trace the entry point and verify that repeated calls produce the same highly_variable gene set; add a regression test covering this behavior and run the relevant test suite.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- bioinformatics
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100