google / google/tensorstore

zarr3 shard user block / offset

Open
#186 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
1.5k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

It would nice if for Zarr version 3 shards, one could specify a byte offset for tensorstore to start writing at. This way one could store additional binary data at the beginning of a zarr shard. This is conceptually similar to a [HDF5 User Block](https://docs.h5py.org/en/stable/high/file.html#user-block). The idea would be to allow an arbitrary number of X bytes at the beginning of a shard file that tensorstore will not touch.

A modification to the Zarr v3 standard is not needed since the offset itself is encoded in the Zarr v3 shard index. This is an implementation option.

With this change the user block could contain HDF5 meta data or TIFF IFDs, allowing Zarr shards to also be HDF5 or Zarr v3 files.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.