developmentseed / developmentseed/zarrista

Extend `EncodedChunk` abstraction to write side

Open
#129 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
45
Forks
2
Avg merge
2d 4h
Merged PRs (30d)
7

Description

Have
```py
Array.encode_chunk() -> EncodedChunk
```
which takes in chunk indices and a typed, decoded array (#47).

Or rather, because chunk encoding will be either sync (current thread) or async (on a Rust thread pool), maybe have a separate `ChunkEncoder` class.

Or, just put encoding onto `CodecChain`, so we have `CodecChain.encode` and `CodecChain.encode_async`.

- We probably want a way to encode/decode chunks that is independent of a store.
- It seems a little messy to put sync/async methods for chunk encoding onto `Array`/ `AsyncArray`, when the sync/async bridge of chunk encoding describes _where the CPU bound work is happening_, not what kind of IO is happening

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the mentioned Array, AsyncArray, EncodedChunk, ChunkEncoder, and CodecChain abstractions and how chunk encoding should remain independent of a store. Resolve whether encoding belongs on a separate encoder or CodecChain, and define the sync and async API that satisfies the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend-api-design, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.