SpikeInterface / SpikeInterface/spikeinterface
Assigning memory usage to sorters
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 847
- Forks
- 280
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 29
Description
Hello,
I am trying to run several spike sorters using spikeinterface. I assign the number of jobs and memory using
global_job_kwargs = dict(n_jobs=12, total_memory='180G')
si.set_global_job_kwargs(**global_job_kwargs)
When I try to run spikesorting using spykingcircus2, using
sorting_spykingcircus2 = ss.run_sorter(sorter_name="spykingcircus2", recording=rec_w,
output_folder="2021-11-29/spkinterface/spykingcircus2",
filtering={},
apply_preprocessing = False,
job_kwargs = {'n_jobs': 12, 'total_memory':'180G'},
detection = {'peak_sign': 'neg', 'detect_threshold': 6},
selection = {"n_peaks_per_channel": 10000,
"min_n_peaks": 20000},
)
I get the error: numpy.core._exceptions._ArrayMemoryError: Unable to allocate 392. GiB for an array with shape (5, 231, 91107520) and data type float32
I tried including matching = {"method": "circus-omp-svd"} within the sorter arguments, but the error persists.
Is there a way to restrict the memory used by spykingcircus or adjust the chunk size? I am using spikeinterface version: 0.99.1.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the si.set_global_job_kwargs and ss.run_sorter entry points, then trace the spykingcircus2 sorter handling for job_kwargs, total_memory, and chunk sizing. Reproduce the reported allocation error with the shown parameters and determine whether the memory limit is honored. Done means the memory behavior is explained and a verified fix or documented limitation is covered by the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100