SpikeInterface / SpikeInterface/spikeinterface

Assigning memory usage to sorters

Open
#2,474 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug in sorter
Dominant language
Python
Stars
847
Forks
280
Avg merge
3d 9h
Merged PRs (30d)
29

Description

Hello,

I am trying to run several spike sorters using spikeinterface. I assign the number of jobs and memory using
global_job_kwargs = dict(n_jobs=12, total_memory='180G')
si.set_global_job_kwargs(**global_job_kwargs)

When I try to run spikesorting using spykingcircus2, using
sorting_spykingcircus2 = ss.run_sorter(sorter_name="spykingcircus2", recording=rec_w,
output_folder="2021-11-29/spkinterface/spykingcircus2",
filtering={},
apply_preprocessing = False,
job_kwargs = {'n_jobs': 12, 'total_memory':'180G'},
detection = {'peak_sign': 'neg', 'detect_threshold': 6},
selection = {"n_peaks_per_channel": 10000,
"min_n_peaks": 20000},
)
I get the error: numpy.core._exceptions._ArrayMemoryError: Unable to allocate 392. GiB for an array with shape (5, 231, 91107520) and data type float32

I tried including matching = {"method": "circus-omp-svd"} within the sorter arguments, but the error persists.
Is there a way to restrict the memory used by spykingcircus or adjust the chunk size? I am using spikeinterface version: 0.99.1.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the si.set_global_job_kwargs and ss.run_sorter entry points, then trace the spykingcircus2 sorter handling for job_kwargs, total_memory, and chunk sizing. Reproduce the reported allocation error with the shown parameters and determine whether the memory limit is honored. Done means the memory behavior is explained and a verified fix or documented limitation is covered by the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.