SpikeInterface / SpikeInterface/spikeinterface

error with running KS4 from spikeinterface

Open
#2,839 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

sorters
Dominant language
Python
Stars
847
Forks
280
Avg merge
3d 9h
Merged PRs (30d)
29

Description

Hi, can you please tell me what is going wrong here?
Trying to call KS4 from spikeinterface, I get the error "slice indices must be integers or None or have an index method".
Please see code and full error report below.

Using:
NPX2.0 (two shanks)
spikeinterface 0.100.6
kilosort 4.0.7

Thanks a lot,
Paola

recording_raw = se.read_openephys(folder_path=path, stream_id="{}".format(index), load_sync_channel=False) 
fs = recording_raw.get_sampling_frequency()

# preprocessing steps
recording = st.correct_lsb(recording_raw, verbose=1);
recording = st.bandpass_filter(recording, freq_min=300, freq_max=6000)
recording = st.phase_shift(recording) 
[bad_channel_ids, channel_labels] = st.detect_bad_channels(recording=recording)  
recording = recording.remove_channels(remove_channel_ids=bad_channel_ids)  

grouped_recordings = recording.split_by(property='group')
recgrouplist_hpsf = [st.highpass_spatial_filter(recording=grouped_recordings[k]) for k in grouped_recordings.keys()]  
recording_hpsf = si.aggregate_channels(recgrouplist_hpsf)
print(recording_hpsf)

print("Starting KS4")
t_start = time.perf_counter()
sorting_ks4_prop = ss.run_sorter_by_property("kilosort4", recording=recording_hpsf, grouping_property="group", working_folder=ks_folder, verbose=True)
t_stop = time.perf_counter()
elapsed_prop = np.round(t_stop - t_start, 2)
print(f"Elapsed time by property: {elapsed_prop} s")

Error running kilosort4

SpikeSortingError Traceback (most recent call last)
Cell In[37], line 117
115 print("Starting KS4")
116 t_start = time.perf_counter()
--> 117 sorting_ks4_prop = ss.run_sorter_by_property("kilosort4", recording=recording_hpsf, grouping_property="group", working_folder=ks_folder, verbose=True)
118 t_stop = time.perf_counter()
119 elapsed_prop = np.round(t_stop - t_start, 2)

File ~\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\sorters\launcher.py:297, in run_sorter_by_property(sorter_name, recording, grouping_property, working_folder, mode_if_folder_exists, engine, engine_kwargs, verbose, docker_image, singularity_image, **sorter_params)
286 job = dict(
287 sorter_name=sorter_name,
288 recording=rec,
(...)
293 **sorter_params,
294 )
295 job_list.append(job)
--> 297 sorting_list = run_sorter_jobs(job_list, engine=engine, engine_kwargs=engine_kwargs, return_output=True)
299 unit_groups = []
300 for sorting, group in zip(sorting_list, recording_dict.keys()):

File ~\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\sorters\launcher.py:106, in run_sorter_jobs(job_list, engine, engine_kwargs, return_output)
103 if engine == "loop":
104 # simple loop in main process
105 for kwargs in job_list:
--> 106 sorting = run_sorter(**kwargs)
107 if return_output:
108 out.append(sorting)

File ~\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\sorters\runsorter.py:175, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
168 container_image = singularity_image
169 return run_sorter_container(
170 container_image=container_image,
171 mode=mode,
172 **common_kwargs,
173 )
--> 175 return run_sorter_local(**common_kwargs)

File ~\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\sorters\runsorter.py:225, in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params)
223 SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose)
224 SorterClass.setup_recording(recording, output_folder, verbose=verbose)
--> 225 SorterClass.run_from_folder(output_folder, raise_error, verbose)
226 if with_output:
227 sorting = SorterClass.get_result_from_folder(output_folder, register_recording=True, sorting_info=True)

File ~\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\sorters\basesorter.py:293, in BaseSorter.run_from_folder(cls, output_folder, raise_error, verbose)
290 print(f"{sorter_name} run time {run_time:0.2f}s")
292 if has_error and raise_error:
--> 293 raise SpikeSortingError(
294 f"Spike sorting error trace:\n{log['error_trace']}\n"
295 f"Spike sorting failed. You can inspect the runtime trace in {output_folder}/spikeinterface_log.json."
296 )
298 return run_time

SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\sorters\basesorter.py", line 258, in run_from_folder
SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\sorters\external\kilosort4.py", line 201, in _run_from_folder
ops = compute_preprocessing(ops, device, tic0=tic0, file_object=file_object)
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\kilosort\run_kilosort.py", line 295, in compute_preprocessing
whiten_mat = preprocessing.get_whitening_matrix(bfile, xc, yc, nskip=nskip,
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\kilosort\preprocessing.py", line 103, in get_whitening_matrix
X = f.padded_batch_to_torch(j)
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\kilosort\io.py", line 716, in padded_batch_to_torch
X = super().padded_batch_to_torch(ibatch)
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\kilosort\io.py", line 518, in padded_batch_to_torch
data = self.file[bstart : bend]
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\kilosort\io.py", line 949, in getitem
samples = self.recording.get_traces(start_frame=i, end_frame=j,
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\core\baserecording.py", line 288, in get_traces
traces = rs.get_traces(start_frame=start_frame, end_frame=end_frame, channel_indices=channel_indices)
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\core\channelslice.py", line 98, in get_traces
traces = self._parent_recording_segment.get_traces(start_frame, end_frame, parent_indices)
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\core\channelsaggregationrecording.py", line 144, in get_traces
traces_recording = segment.get_traces(
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\preprocessing\highpass_spatial_filter.py", line 189, in get_traces
traces, left_margin, right_margin = get_chunk_with_margin(
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\core\recording_tools.py", line 721, in get_chunk_with_margin
traces_chunk = rec_segment.get_traces(
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\core\channelslice.py", line 98, in get_traces
traces = self._parent_recording_segment.get_traces(start_frame, end_frame, parent_indices)
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\core\channelslice.py", line 98, in get_traces
traces = self._parent_recording_segment.get_traces(start_frame, end_frame, parent_indices)
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\preprocessing\phase_shift.py", line 92, in get_traces
traces_chunk, left_margin, right_margin = get_chunk_with_margin(
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\core\recording_tools.py", line 751, in get_chunk_with_margin
traces_chunk = rec_segment.get_traces(start_frame2, end_frame2, channel_indices)
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\preprocessing\filter.py", line 132, in get_traces
traces_chunk, left_margin, right_margin = get_chunk_with_margin(
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\core\recording_tools.py", line 721, in get_chunk_with_margin
traces_chunk = rec_segment.get_traces(
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\preprocessing\normalize_scale.py", line 27, in get_traces
scaled_traces = self.parent_recording_segment.get_traces(start_frame, end_frame, channel_indices).astype(
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\preprocessing\normalize_scale.py", line 27, in get_traces
scaled_traces = self.parent_recording_segment.get_traces(start_frame, end_frame, channel_indices).astype(
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\spikeinterface\extractors\neoextractors\neobaseextractor.py", line 326, in get_traces
raw_traces = self.neo_reader.get_analogsignal_chunk(
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\neo\rawio\baserawio.py", line 805, in get_analogsignal_chunk
raw_chunk = self._get_analogsignal_chunk(block_index, seg_index, i_start, i_stop, stream_index, channel_indexes)
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\neo\rawio\openephysbinaryrawio.py", line 385, in _get_analogsignal_chunk
sigs = sigs[i_start:i_stop, :]
File "C:\Users\SNeurobiology\miniconda3\envs\ephys-env\lib\site-packages\numpy\core\memmap.py", line 335, in getitem
res = super().getitem(index)
TypeError: slice indices must be integers or None or have an index method

Spike sorting failed. You can inspect the runtime trace in E:\paola_temp\E1_M15\NPXData\2024-04-12_17-49-04\Record Node 101\experiment1\recording1\SpikeData_ProbeA\0/spikeinterface_log.json.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported run_sorter_by_property call with spikeinterface 0.100.6 and kilosort 4.0.7, then trace the failure from kilosort/io.py through spikeinterface/core/channelsaggregationrecording.py to neo/rawio/openephysbinaryrawio.py. Done means the supplied Open Ephys pipeline no longer raises the slice-index TypeError and the failing path has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.