SpikeInterface / SpikeInterface/spikeinterface
Migrating to zarr v3
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 847
- Forks
- 280
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 29
Description
Currently spike interface uses zarr v2 which does not have a compatible API with zarr v3 which has different features (mainly sharding).
My motivation behind sharding is that I like being able to load a single channel pretty fast, so I usually put channel_chunk_size fairly low, but by doing so, I get a huge number of files (384x2x3600 for a 2h recording...).
I've tried to migrate the code of zarrextractors.py and here is my conclusion:
- migrating to v3 is mostly easy by changing
group.create_dataset(key=name,..., data=data)bygroup.create_array(key=name, ..., shape=data.shape, dtype=data.dtype)followed bygroup[name][:] = data - we have a problem for structured numpy arrays which were handled in v2 and not in v3. These seem in the process of being handled by zarr, see this pr. Either we wish to rely on this future implementation or we can have a custom convention for handling them. A solution could be to create a group with a name like "_structuredarraygrp[name]" and put each individual array in it.
- However, for reading the structured arrays, the code is less obvious: either we remove the lazyness of zarr and we just create the numpy array from the group each time, or we need to find a way to provide a lazyarray that handles structured dtypes. Perhaps by using dask ?
- adding sharding is extremely easy as its just a parameter of create_array. However, one needs to modify the functions that process arguments.
Is there any interest in such a migration (enough for me to submit a pr) ?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing zarrextractors.py and the functions that process arguments for array creation. Investigate zarr v3 support for structured arrays and lazy reads, including the linked zarr PR, then determine how sharding should be exposed. Done means the migration strategy is resolved and the relevant read and write paths support the required behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100