scverse / scverse/scanpy

"batch_size" parameter doesn't work for "sc.external.pp.scanorama_integrate"

Open
#2,319 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.6k
Forks
779
Avg merge
1d 4h
Merged PRs (30d)
27

Description

According to scanorama MemoryError, I think batch_size parameter doesn't work for sc.external.pp.scanorama_integrate

Minimal code sample (that we can copy&paste without having any data)
sc.external.pp.scanorama_integrate(adata, key="datasetID", batch_size=500)
WARNING: Out of memory, consider turning on batched computation with batch_size parameter.
Traceback (most recent call last):
  File "/lustre1/shiq//02_igt/py/1_igt_scanorama.py", line 26, in <module>
    sc.external.pp.scanorama_integrate(adata, key="datasetID", batch_size=500)
  File "/lustre1/shiq/app/miniconda3/envs/scanpy/lib/python3.10/site-packages/scanpy/external/pp/_scanorama_integrate.py", line 121, in scanorama_integrate
    integrated = scanorama.assemble(
  File "/lustre1/shiq/app/miniconda3/envs/scanpy/lib/python3.10/site-packages/scanorama/scanorama.py", line 933, in assemble
    bias = transform(curr_ds, curr_ref, ds_ind, ref_ind, sigma=sigma,
  File "/lustre1/shiq/app/miniconda3/envs/scanpy/lib/python3.10/site-packages/scanorama/scanorama.py", line 762, in transform
    avg_bias = batch_bias(curr_ds, match_ds, bias, sigma=sigma,
  File "/lustre1/shiq/app/miniconda3/envs/scanpy/lib/python3.10/site-packages/scanorama/scanorama.py", line 723, in batch_bias
    weights = rbf_kernel(curr_ds, match_ds, gamma=0.5*sigma)
  File "/lustre1/shiq/app/miniconda3/envs/scanpy/lib/python3.10/site-packages/sklearn/metrics/pairwise.py", line 1294, in rbf_kernel
    X, Y = check_pairwise_arrays(X, Y)
  File "/lustre1/shiq/app/miniconda3/envs/scanpy/lib/python3.10/site-packages/sklearn/metrics/pairwise.py", line 155, in check_pairwise_arrays
    X = check_array(
  File "/lustre1/shiq/app/miniconda3/envs/scanpy/lib/python3.10/site-packages/sklearn/utils/validation.py", line 727, in check_array
    warnings.warn(
FutureWarning: np.matrix usage is deprecated in 1.0 and will raise a TypeError in 1.2. Please convert to a numpy array with np.asarray. For more information see: https://numpy.org/doc/stable/reference/generated/numpy.matrix.html

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read scanpy/external/pp/_scanorama_integrate.py, especially the scanorama_integrate entry point and its scanorama.assemble call, then trace batch_size into the dependency’s assemble/transform path. Reproduce the provided call and traceback; done means the parameter reaches the intended batched computation without the reported out-of-memory path.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, scikit-learn
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.