lance-format / lance-format/lance

Expose a public toggle for disable_transaction_file (skip redundant external transaction file)

Open
#7,486 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Background

Every commit currently writes the transaction in two places:

  • Inline in the manifest (transaction_section) — written unconditionally. do_write_manifest sets manifest.transaction_section whenever a transaction is passed (rust/lance-table/src/io/manifest.rs:153-159), and all commit paths pass Some(transaction) (rust/lance/src/io/commit.rs:265,842,1047).
  • A separate _transactions/ file — written whenever !disable_transaction_file() (rust/lance/src/io/commit.rs:151,786,981).

FLAG_DISABLE_TRANSACTION_FILE (bit 32, rust/lance-table/src/feature_flags.rs:22) and the commit-path support already exist to skip the now-redundant external file. But disable_transaction_file lives only on the internal pub(crate) struct ManifestWriteConfig (rust/lance/src/dataset.rs:3438) and is never set to true anywhere — no WriteParams/CommitConfig field, no Python/Java binding, no env var, no config key, not even in tests.

So the optimization is plumbed end-to-end except for the one switch that turns it on, and we pay one redundant write per commit.

Ask

Wire up a public way to enable disable_transaction_file:

  • Plumb it through a user-facing surface (e.g. WriteParams / CommitConfig, with Python and Java bindings kept as thin wrappers), keeping the parameter name consistent across languages.
  • Add tests covering: a commit with the external file disabled produces a readable dataset from the inline transaction_section, and the writer feature flag (bit 32) is set.

Note: enabling it sets a writer feature flag, so older readers that don't understand bit 32 will reject the dataset. Defaulting it on is tracked separately (see follow-up issue).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing ManifestWriteConfig in rust/lance/src/dataset.rs into the commit paths in rust/lance/src/io/commit.rs, then inspect the flag definition in rust/lance-table/src/feature_flags.rs and transaction handling in rust/lance-table/src/io/manifest.rs. Identify the user-facing write configuration and its Python and Java wrappers. Done means callers can enable the option, a commit remains readable from the inline transaction_section, and writer feature flag bit 32 is set.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python, rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.