developmentseed / developmentseed/zarrista
define a chunked array protocol
- Dominant language
- Python
- Stars
- 45
- Forks
- 2
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 7
Description
dunders like `__array__` and `__dlpack__` are useful for extracting an array into a numpy array or a dlpack capsule. I'm not aware of a community-supported analog for extracting a chunked array into a chunked array container, like a dask array or a cubed array. This means we need to define an abstract chunked array, and then define a protocol that an implementation of that abstract chunked array would use to instantiate a real chunked array instance.
A few of my assumptions about the abstract chunked array:
- each chunk has the same data type
- the chunks are represented on a finite rectilinear grid (regular is a special case)
- each chunk decodes to the same in-memory array type(s)
- the grid of chunks must not be dense, so a fill value is required
As data, these assumptions are basically a subset of the Zarr array metadata document, with information about what in-memory array(s) the chunks decode into (e.g., which dunder methods does each decoded chunk define).
So if If we define `__chunked_array__` and `__async_chunked_array__` protocols for an object that decodes into an arbitrary chunked array, what should their signatures be? Something like `__chunked_array__(self) -> Mapping[tuple[int, ...], Callable[[], ArrayLike]]`, i.e. it returns a mapping from chunk indices to functions that produce chunks? And for the async case, the callable would return a coroutine?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the issue's assumptions about abstract chunked arrays and the proposed __chunked_array__ and __async_chunked_array__ signatures. Compare the mentioned __array__ and __dlpack__ protocols and the Zarr array metadata model, then establish agreed synchronous and asynchronous protocol contracts. Done means the abstract representation and both protocol signatures are clearly specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- api, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100