SpikeInterface / SpikeInterface/spikeinterface
export_report is very slow
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 847
- Forks
- 280
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 29
Description
Hi there — thank you all for maintaining this suite of great tools! I am trying to write a script that handles pre-processing / sorting / post-processing all at once.
Describe the issue
I've noticed that the export_report function is very slow — almost 30 seconds per unit. If I have a few hundred units, that's an extra hour, which is in some cases more than the entire pre-processing + sorting!
My recording is 1 hour long, and the units seem to have relatively low firing rates (1 - 20) so I don't think that's the issue. This is on version 0.100.4, so I'm willing to be told that these issues have been solved, but just poking around the code a bit, it doesn't look like it.
Reproducing
I can reproduce this by simply re-loading the waveform extractor from a folder like we = si.load_waveforms(waveform_dir), and then running export_report(we, output_folder=qc_dir, **job_kwargs). Everything is fast except generating the per-unit plots at the end.
Ideas
I dug around a bit, and I have two clues:
- the slowest part seems to be
sw.plot_unit_waveforms, which takes about ~20 seconds on its own. I can't figure out why it's slow, as loading in the templates / waveforms (i.e.we.get_waveforms(60)) is very fast. I guess matplotlib is slow to plot hundreds of lines? Speaking for myself, I find the smear of all the raw waveforms totally uninformative:
- for the amplitudes part of the plot: running something like
sw.plot_amplitudes(we, unit_ids=[60])directly, takes roughly the same amount of time (~13 seconds) as just loading the data with this line: https://github.com/SpikeInterface/spikeinterface/blob/7d0e1da655beddb414c21a3a0b2d65b9ab115f1d/src/spikeinterface/widgets/amplitudes.py#L56
I see that the point of loading all the data is to allow the widget to plot arbitrarily many units at once, but in the case where we plot only one unit per instance of the widget, many times, this is a huge time suck. I admit I don't really see why loading the amplitude data takes so long, if I just runnp.load("/path/to/amplitude_segment_0.npy")it's very fast (< 1 sec).
So in summary, I would suggest:
-- have an option to plot mean/std instead of raw waveforms, or just not show waveforms at all, in the export_report function.
-- figure out a way to make loading unit amplitudes as fast as loading the waveform data.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with export_report and profile the per-unit calls to sw.plot_unit_waveforms and sw.plot_amplitudes; inspect amplitudes.py around line 56, where amplitude data is loaded. Compare report generation with direct waveform and amplitude loading. Done means the report avoids unnecessary repeated work or provides an option to omit or simplify waveform plots, with a measurable reduction in per-unit runtime.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- matplotlib, numpy, python
- Domain
- data-visualization, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100