lance-format / lance-format/lance

feat(java): expose mergeInsertUncommitted (execute_uncommitted) in Java SDK

Open
#8,546 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Problem

The Java SDK's Dataset.mergeInsert() always commits immediately. There is no way to perform a merge-insert and hold the resulting Transaction for manual commit via CommitBuilder.

Rust core already supports this via MergeInsertJob::execute_uncommitted() and MergeInsertJob::execute_uncommitted_batches(), and the Python SDK already exposes MergeInsertBuilder.execute_uncommitted() and execute_uncommitted_batches().

Use Case

Callers need to inspect or mutate the uncommitted Transaction (e.g., add extra operations, validate before commit) before finalizing, rather than having the merge-insert auto-commit.

Current State

Layer mergeInsert (auto-commit) mergeInsertUncommitted
Rust core Yes Yes
Python SDK Yes Yes
Java SDK Yes No

What's Missing

All the building blocks are already in place — CommitBuilder, Transaction, MergeInsertParams, MergeInsertStats all exist in Java. The gap is small:

  1. JNI (java/lance-jni/src/merge_insert.rs): New JNI function calling execute_uncommitted instead of execute_reader, returning (Transaction, MergeStats) across the JNI boundary.

  2. Java (java/src/main/java/org/lance/):

    • New UncommittedMergeInsertResult POJO holding Dataset, Transaction, MergeInsertStats
    • New Dataset.mergeInsertUncommitted(MergeInsertParams, ArrowArrayStream) method
    • Corresponding native declaration

Estimated total: ~150 lines of binding code. No new Rust core logic needed.

Proposed API

UncommittedMergeInsertResult result = dataset.mergeInsertUncommitted(
    new MergeInsertParams(List.of("id"))
        .withMatchedUpdateAll()
        .withNotMatched(WhenNotMatched.InsertAll),
    sourceStream
);

// Inspect / modify transaction before commit
Transaction txn = result.transaction();
Dataset committed = new CommitBuilder(result.dataset()).execute(txn);

References

  • Rust: rust/lance/src/dataset/write/merge_insert.rsexecute_uncommitted() / execute_uncommitted_batches()
  • Python binding: python/src/dataset.rsMergeInsertBuilder::execute_uncommitted() (line 468)
  • Existing Java mergeInsert: java/src/main/java/org/lance/Dataset.java (line 2154)
  • Existing Java JNI: java/lance-jni/src/merge_insert.rs

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with java/src/main/java/org/lance/Dataset.java and java/lance-jni/src/merge_insert.rs, then compare the Rust execute_uncommitted APIs with the Python binding in python/src/dataset.rs. Done means the Java SDK exposes the uncommitted result, its native declaration, and JNI support for returning the Transaction and merge statistics so callers can commit manually.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, rust
Domain
api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
63/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.