SpikeInterface / SpikeInterface/spikeinterface
Sorting analyzer sometimes does not work
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 847
- Forks
- 280
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 29
Description
Hi. My spikes are sorted with kilosort4 on a neuropixel 1 probe. Sorting analyzer works for some sessions but not for others. I get the error:
estimate_sparsity (workers: 10 processes): 100%|██████████| 1830/1830 [06:19<00:00, 4.83it/s]
compute_waveforms (workers: 10 processes): 100%|██████████| 1830/1830 [07:11<00:00, 4.24it/s]
noise_level (workers: 10 processes): 100%|██████████| 20/20 [00:00<00:00, 34.29it/s]
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\spawn.py", line 132, in _main
self = reduction.pickle.load(from_parent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_pickle.UnpicklingError: pickle data was truncated
Traceback (most recent call last):
File "", line 1, in
File "C:\Program Files\JetBrains\PyCharm 2024.3.2\plugins\python-ce\helpers\pydev_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\JetBrains\PyCharm 2024.3.2\plugins\python-ce\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "Z:\M1_thalamus\tasksignals\sorting_analyzer.py", line 39, in
main()
File "Z:\M1_thalamus\tasksignals\sorting_analyzer.py", line 34, in main
sorting_analyzer.compute(['random_spikes', 'templates', 'spike_amplitudes', 'spike_locations', 'waveforms', 'noise_levels',
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\sortinganalyzer.py", line 1461, in compute
self.compute_several_extensions(extensions=extensions, save=save, verbose=verbose, **job_kwargs)
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\sortinganalyzer.py", line 1614, in compute_several_extensions
results = run_node_pipeline(
^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\node_pipeline.py", line 631, in run_node_pipeline
processor.run(recording_slices=recording_slices)
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\job_tools.py", line 517, in run
results = executor.map(process_function_wrapper, recording_slices)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 851, in map
results = super().map(partial(_process_chunk, fn),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures_base.py", line 608, in map
fs = [self.submit(fn, *args) for args in zip(*iterables)]
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 822, in submit
self._adjust_process_count()
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 781, in _adjust_process_count
self._spawn_process()
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 799, in _spawn_process
p.start()
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\context.py", line 337, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\popen_spawn_win32.py", line 95, in init
reduction.dump(process_obj, to_child)
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
OSError: [Errno 22] Invalid argument
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with sorting_analyzer.py at lines 34 and 39, then trace the call into spikeinterface/core/sortinganalyzer.py, node_pipeline.py, and job_tools.py. Reproduce the analyzer computation on Windows using the reported sessions and multiprocessing settings; done means sorting analyzer completes reliably without the reported pickle or invalid-argument errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100