developmentseed / developmentseed/zarrista

Is it a problem to reconstruct & have two icechunk sessions

Open
#107 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
45
Forks
2
Avg merge
2d 4h
Merged PRs (30d)
7

Description

It's not possible to share a Rust object across two Rust-Python libraries that were compiled independently.

This means that for Zarrista to natively work with icechunk as a store (where zarrista manages icechunk from Rust without going through Python) Zarrista has to serialize the icechunk state from the icechunk python library and create a new icechunk session in the statically-compiled icechunk version in zarrista.

Unlike obstore, Icechunk is not stateless, so perhaps there could be bad things that happen if two different sessions are trying to work on the same "repo" at the same time?

I think this session serialization may be the same process done when an icechunk session is pickled, so _hopefully_ it won't be a problem, but I don't know.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Begin by tracing the session serialization used between the icechunk Python library and zarrista's statically compiled Rust version, then compare it with icechunk session pickling. Done means establishing whether reconstructed sessions can safely operate on the same repository concurrently and recording the conclusion or required follow-up.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.