lance-format / lance-format/lance
How can I implement a single commit handler when writing to a dataset using both Rust and Python simultaneously?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
My application scenario involves writing datasets using Python and Rust. I implemented an External Manifest Store in Rust, which provides atomic transaction commits. However, this solution cannot be used with Python, and the commit handler has requirements for the writers, specifically that only one commit handler can be active at a time.Are there any solutions to this problem?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the Rust External Manifest Store and the Python dataset-writing path, focusing on how commit handlers are created and enforced for writers. Determine whether the existing interfaces expose a shared cross-language transaction mechanism. Done means identifying and documenting a supported way for Rust and Python writers to coordinate a single active commit handler, or clearly recording the missing capability.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100