dandi / dandi/dandi-cli

Partial Zarr Directory Updates in Dandi and LINC

Open
#1,474 3 comments 3 reactions 0 assignees View on GitHub
zarr
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

Cc @dstansby @kabilar @satra @yarikoptic @waxlamp @balbasty

In the LINC project, @dstansby encountered a scenario where an update was requested for a portion of a Zarr directory.
Currently, DANDI and LINC treat a Zarr directory as a single object tree, requiring the entire directory to be downloaded even for updates that only modify specific pieces.

Downloading the entire Zarr directory can be inefficient, especially for large datasets where only a small portion needs updating.

This issue's purpose is to capture the need for mechanism to allow for partial updates of Zarr directories within Dandi and LINC.

Analagous, @satra suggested the initial usage of [zarrita](https://github.com/scalableminds/zarrita) to explore elements of sharding, with perhaps the LINC project as a place to test

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.