SpikeInterface / SpikeInterface/spikeinterface

export_to_phy vs using kilosort sorter output params.py directly

Open
#4,635 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

exporters
Dominant language
Python
Stars
847
Forks
280
Avg merge
3d 9h
Merged PRs (30d)
29

Description

Hello!
We realised in our lab recently that two of us were using two different ways to manually curate units with Phy after sorting with kilosort4.
The first is to use to use the export_to_phy function and then open the params.py file with Phy as described in the documentation (the exporting is very slow, sometimes slower than the actual sorting):

analyzer = si.create_sorting_analyzer(sorting_KS4, recording_saved, sparse=True)
# compute all the extensions required
sexp.export_to_phy(sorting_analyzer=analyzer, output_folder=Path(data_dir) / 'phy_folder', verbose=True, copy_binary=False)

The second is to just compute the extensions needed, save them as tsv files, copy to the sorter output location where params.py from the sorter output is generated, and then just open that with Phy without the export_to_phy function (much faster, barely any extra compute time).

sorting_analyzer = si.create_sorting_analyzer(sorting=sorting_KS4, recording=rec_corrected, format="binary_folder", folder = KSfolder / 'analyzer_med' )
contamination = sqm.compute_sliding_rp_violations(sorting_analyzer=sorting_analyzer,
                                                  bin_size_ms=0.25)

presence_ratio = sqm.compute_presence_ratios(sorting_analyzer=sorting_analyzer)

def save_dict_to_tsv(data, header_name, file_path, delimiter='\t'):
    """
    Saves a dictionary to a TSV file.

    Args:
        data (dict): The dictionary to save. Keys will be the header row.
        file_path (str): The path to the TSV file.
        delimiter (str, optional): The delimiter. Defaults to tab ('\t').
    """
    #with open(file_path, 'w', newline='', encoding='utf-8') as tsvfile:
    with open(Path(KSfolder) / 'sorter_output' / file_path, 'w', newline='', encoding='utf-8') as tsvfile:

        writer = csv.writer(tsvfile, delimiter=delimiter)

        writer.writerow(['cluster_id', header_name])
        for key, value in data.items():
            writer.writerow([key, value])

save_dict_to_tsv(contamination, 'sliding_rp', 'cluster_sliding_rp.tsv')
save_dict_to_tsv(presence_ratio, 'presence', 'cluster_presence.tsv')
# move these to the same folder that holds your params.py file for phy


We tried both for the same sorting, and the only thing that jumped out to us was that the second method resulted in fewer channels in the waveform view on Phy, but no other noticeable difference.

What does export_to_phy do that takes so much time, and is it necessary to do it, since the second method seems to be working fine? Or are we missing something here?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the export_to_phy function and compare the files it produces with the params.py and TSV files used by the direct Kilosort4 workflow. Reproduce both workflows on the same sorting, then identify what extra work accounts for the runtime and fewer waveform channels, and document whether the direct workflow is equivalent.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.