developmentseed / developmentseed/zarrista

define a chunked array protocol

Open
#100 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
45
Forks
2
Avg merge
2d 4h
Merged PRs (30d)
7

Description

dunders like `__array__` and `__dlpack__` are useful for extracting an array into a numpy array or a dlpack capsule. I'm not aware of a community-supported analog for extracting a chunked array into a chunked array container, like a dask array or a cubed array. This means we need to define an abstract chunked array, and then define a protocol that an implementation of that abstract chunked array would use to instantiate a real chunked array instance.

A few of my assumptions about the abstract chunked array:
- each chunk has the same data type
- the chunks are represented on a finite rectilinear grid (regular is a special case)
- each chunk decodes to the same in-memory array type(s)
- the grid of chunks must not be dense, so a fill value is required

As data, these assumptions are basically a subset of the Zarr array metadata document, with information about what in-memory array(s) the chunks decode into (e.g., which dunder methods does each decoded chunk define).

So if If we define `__chunked_array__` and `__async_chunked_array__` protocols for an object that decodes into an arbitrary chunked array, what should their signatures be? Something like `__chunked_array__(self) -> Mapping[tuple[int, ...], Callable[[], ArrayLike]]`, i.e. it returns a mapping from chunk indices to functions that produce chunks? And for the async case, the callable would return a coroutine?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the issue's assumptions about abstract chunked arrays and the proposed __chunked_array__ and __async_chunked_array__ signatures. Compare the mentioned __array__ and __dlpack__ protocols and the Zarr array metadata model, then establish agreed synchronous and asynchronous protocol contracts. Done means the abstract representation and both protocol signatures are clearly specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
api, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.