Zarrista overview & goals

Human written by @kylebarron

## Background

Zarr is the pre-eminent data format for storing N-dimensional data. [Zarr-Python](https://zarr.readthedocs.io/en/stable/) is the primary Python library for reading/writing this data; [Zarrs](https://github.com/zarrs/zarrs) is the primary Rust library for reading/writing Zarr, and is potentially much faster than Zarr-Python.

@d-v-b recently prototyped vibe-coded Zarrs bindings for Zarr-Python in https://github.com/zarr-developers/zarr-python/pull/4064. After some discussion, I decided to start prototyping a standalone Zarrs binding for Python, with the goal of providing another option for https://github.com/zarr-developers/zarr-python/pull/4064. That is, Zarrista should be a low-level binding which Zarr-Python could potentially wrap in its higher level APIs in the future.

@d-v-b found that https://github.com/zarr-developers/zarr-python/pull/4064 had vastly improved performance:

> I'm seeing ~15x throughput improvement, looks good.

This gives strong motivation for the potential performance improvements of a Python Zarr library built on Zarrs.

## Goals

Zarrista should be both directly usable by intermediate-to-advanced users, but should also be built so that Zarr-Python could theoretically build on top of it in the future.

Zarrista should not add logic beyond what already exists in Zarrs. Similar to [Obstore](https://github.com/developmentseed/obstore), the scope should be limited to _only_ what is already implemented upstream. This keeps maintainability high.

Zarrista should expose as many APIs as possible from Zarrs. `Array` and `Group` will be medium-high level APIs, but ideally all lower level APIs (if stable) should also be exposed, so that downstream libraries can choose the most performance route for them.

- Create a minimal but complete Python binding of Zarrs
- [x] Include parallel sync and async stores, arrays, and groups
- [x] Support chunk reading
- [x] Support array reading
- [ ] Support chunk writing
- [ ] Support array writing
- Various store support
- [x] Icechunk integration
- [x] ObjectStore/Obstore integration
- [x] Filesystem integration
- [x] Memory store integration (for testing only)
- Zero copy data exchange between Rust and Python
- Primitive, fixed width types:
- [x] Buffer protocol/numpy
- [ ] Arrow
- [x] DLPack
- Variable width types:
- [ ] Buffer protocol/numpy
- [x] Arrow
- [ ] DLPack
- Masked types:
- [ ] Buffer protocol/numpy
- [ ] Arrow
- [ ] DLPack

## Partner and Stakeholders

Partner with @d-v-b as needed for general project direction, and to ensure that public-facing API is usable

Keeping the scope limited to what is already implemented upstream keeps this project focused to what my (@kylebarron's) strengths are. Given my experience from [obstore](https://github.com/developmentseed/obstore), [async-tiff](https://github.com/developmentseed/async-tiff), and similar projects, I'm really good at creating high-performance, Pythonic APIs from Rust libraries. I have considerably less experience with _Zarr itself_. This means that I'm ill-equipped to design a higher-level Zarr API myself; that task should be left to @d-v-b and others in higher-level wrapper libraries (Zarr-Python or other).

## Milestones

cc @developmentseed/cng-island