SpikeInterface / SpikeInterface/spikeinterface

Instability with SC2 in dense recordings

Open
#4,150 17 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

sorters
Dominant language
Python
Stars
847
Forks
280
Avg merge
3d 9h
Merged PRs (30d)
29

Description

Sorry to bring this up again, as I'm pretty sure this is only an issue on recordings with a few thousands of units.

I did not keep up with new commits for a while, and when trying to run sorting again on the latest version it systematically crashes.

The weird thing is that the point at which it crashes is not always reproducible. Starting the script from a local session, over SSH or in a tmux server also seems to have an influence.

The most common crash points are :

  • Creating the mask array in compute_similarity_with_templates_array (here)
  • During the main loop of _compute_similarity_matrix_numba (here)

Looking at the resource usage, the pattern is always the same : there are several Python processes, the number of which is equal to n_jobs, with an equal amount of memory usage, and another Python process with a lot more memory (20-30GB). Then, that amount rapidly climbs to to ~50-60GB, then the session crashes. I'm not entirely sure this is an OOM error, because this amount is still small enough to fit in memory. So either it rises so fast that the session crashes before the display has time to update, or something else is happening.

As a side note, _get_optimal_n_jobs works properly and n_jobs is scaled to memory appropriately. I've noticed that it is only used when templates_from_svd is set to False but I'm not quite sure of the difference.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/spikeinterface/postprocessing/template_similarity.py at compute_similarity_with_templates_array and _compute_similarity_matrix_numba, then reproduce the crash with a dense SC2 recording. Compare runs with templates_from_svd enabled or disabled and inspect the n_jobs and memory behavior described in the issue. Done means sorting no longer crashes under the reported dense-recording conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.