zarr3 shard user block / offset
- Dominant language
- C++
- Stars
- 1.5k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
It would nice if for Zarr version 3 shards, one could specify a byte offset for tensorstore to start writing at. This way one could store additional binary data at the beginning of a zarr shard. This is conceptually similar to a [HDF5 User Block](https://docs.h5py.org/en/stable/high/file.html#user-block). The idea would be to allow an arbitrary number of X bytes at the beginning of a shard file that tensorstore will not touch.
A modification to the Zarr v3 standard is not needed since the offset itself is encoded in the Zarr v3 shard index. This is an implementation option.
With this change the user block could contain HDF5 meta data or TIFF IFDs, allowing Zarr shards to also be HDF5 or Zarr v3 files.
Contributor guide
Assessment
This issue has not been assessed yet.