SpikeInterface / SpikeInterface/spikeinterface
GatherToNpy/Zarr could pre-allocate stores when number of spike is known
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 847
- Forks
- 280
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 29
Description
See https://github.com/SpikeInterface/spikeinterface/pull/4703#discussion_r4024420813
Currently the GatherToNpy and GatherToZarr sequentially write chunk of pipeline data as they are available. While this is needed when we don't know the number of peaks per chunk, in cases when this is known (ie. the pipeline has a PeakRetriever or SpikeRetriever), the gather classes could pre-allocate the entire arrays and write to the exact location. This would speed things up especially for zarr, since the current implementation might require a lot of rechunking under the hood
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked pull request discussion and the GatherToNpy and GatherToZarr implementations, then inspect how PeakRetriever and SpikeRetriever expose known spike counts. Done means known-size pipelines pre-allocate their arrays and write to exact locations, while unknown-size pipelines retain sequential chunk writing; verify the resulting performance and storage behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100