SpikeInterface / SpikeInterface/spikeinterface

Sorting analyzer sometimes does not work

Open
#4,050 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

concurrency
Dominant language
Python
Stars
847
Forks
280
Avg merge
3d 9h
Merged PRs (30d)
29

Description

Hi. My spikes are sorted with kilosort4 on a neuropixel 1 probe. Sorting analyzer works for some sessions but not for others. I get the error:

estimate_sparsity (workers: 10 processes): 100%|██████████| 1830/1830 [06:19<00:00, 4.83it/s]
compute_waveforms (workers: 10 processes): 100%|██████████| 1830/1830 [07:11<00:00, 4.24it/s]
noise_level (workers: 10 processes): 100%|██████████| 20/20 [00:00<00:00, 34.29it/s]
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\spawn.py", line 132, in _main
self = reduction.pickle.load(from_parent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_pickle.UnpicklingError: pickle data was truncated
Traceback (most recent call last):
File "", line 1, in
File "C:\Program Files\JetBrains\PyCharm 2024.3.2\plugins\python-ce\helpers\pydev_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\JetBrains\PyCharm 2024.3.2\plugins\python-ce\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "Z:\M1_thalamus\tasksignals\sorting_analyzer.py", line 39, in
main()
File "Z:\M1_thalamus\tasksignals\sorting_analyzer.py", line 34, in main
sorting_analyzer.compute(['random_spikes', 'templates', 'spike_amplitudes', 'spike_locations', 'waveforms', 'noise_levels',
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\sortinganalyzer.py", line 1461, in compute
self.compute_several_extensions(extensions=extensions, save=save, verbose=verbose, **job_kwargs)
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\sortinganalyzer.py", line 1614, in compute_several_extensions
results = run_node_pipeline(
^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\node_pipeline.py", line 631, in run_node_pipeline
processor.run(recording_slices=recording_slices)
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\job_tools.py", line 517, in run
results = executor.map(process_function_wrapper, recording_slices)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 851, in map
results = super().map(partial(_process_chunk, fn),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures_base.py", line 608, in map
fs = [self.submit(fn, *args) for args in zip(*iterables)]
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 822, in submit
self._adjust_process_count()
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 781, in _adjust_process_count
self._spawn_process()
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 799, in _spawn_process
p.start()
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\context.py", line 337, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\popen_spawn_win32.py", line 95, in init
reduction.dump(process_obj, to_child)
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
OSError: [Errno 22] Invalid argument

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with sorting_analyzer.py at lines 34 and 39, then trace the call into spikeinterface/core/sortinganalyzer.py, node_pipeline.py, and job_tools.py. Reproduce the analyzer computation on Windows using the reported sessions and multiprocessing settings; done means sorting analyzer completes reliably without the reported pickle or invalid-argument errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.