SpikeInterface / SpikeInterface/spikeinterface

Migrating to zarr v3

Open
#4,014 10 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

core
Dominant language
Python
Stars
847
Forks
280
Avg merge
3d 9h
Merged PRs (30d)
29

Description

Currently spike interface uses zarr v2 which does not have a compatible API with zarr v3 which has different features (mainly sharding).
My motivation behind sharding is that I like being able to load a single channel pretty fast, so I usually put channel_chunk_size fairly low, but by doing so, I get a huge number of files (384x2x3600 for a 2h recording...).

I've tried to migrate the code of zarrextractors.py and here is my conclusion:

  • migrating to v3 is mostly easy by changing group.create_dataset(key=name,..., data=data) by group.create_array(key=name, ..., shape=data.shape, dtype=data.dtype) followed by group[name][:] = data
  • we have a problem for structured numpy arrays which were handled in v2 and not in v3. These seem in the process of being handled by zarr, see this pr. Either we wish to rely on this future implementation or we can have a custom convention for handling them. A solution could be to create a group with a name like "_structuredarraygrp[name]" and put each individual array in it.
  • However, for reading the structured arrays, the code is less obvious: either we remove the lazyness of zarr and we just create the numpy array from the group each time, or we need to find a way to provide a lazyarray that handles structured dtypes. Perhaps by using dask ?
  • adding sharding is extremely easy as its just a parameter of create_array. However, one needs to modify the functions that process arguments.

Is there any interest in such a migration (enough for me to submit a pr) ?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing zarrextractors.py and the functions that process arguments for array creation. Investigate zarr v3 support for structured arrays and lazy reads, including the linked zarr PR, then determine how sharding should be exposed. Done means the migration strategy is resolved and the relevant read and write paths support the required behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.