lance-format / lance-format/lance

How can I implement a single commit handler when writing to a dataset using both Rust and Python simultaneously?

Open
#5,751 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-python question rust
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

My application scenario involves writing datasets using Python and Rust. I implemented an External Manifest Store in Rust, which provides atomic transaction commits. However, this solution cannot be used with Python, and the commit handler has requirements for the writers, specifically that only one commit handler can be active at a time.Are there any solutions to this problem?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the Rust External Manifest Store and the Python dataset-writing path, focusing on how commit handlers are created and enforced for writers. Determine whether the existing interfaces expose a shared cross-language transaction mechanism. Done means identifying and documenting a supported way for Rust and Python writers to coordinate a single active commit handler, or clearly recording the missing capability.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.