lance-format / lance-format/lance
Expose a public toggle for disable_transaction_file (skip redundant external transaction file)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Background
Every commit currently writes the transaction in two places:
- Inline in the manifest (
transaction_section) — written unconditionally.do_write_manifestsetsmanifest.transaction_sectionwhenever a transaction is passed (rust/lance-table/src/io/manifest.rs:153-159), and all commit paths passSome(transaction)(rust/lance/src/io/commit.rs:265,842,1047). - A separate
_transactions/file — written whenever!disable_transaction_file()(rust/lance/src/io/commit.rs:151,786,981).
FLAG_DISABLE_TRANSACTION_FILE (bit 32, rust/lance-table/src/feature_flags.rs:22) and the commit-path support already exist to skip the now-redundant external file. But disable_transaction_file lives only on the internal pub(crate) struct ManifestWriteConfig (rust/lance/src/dataset.rs:3438) and is never set to true anywhere — no WriteParams/CommitConfig field, no Python/Java binding, no env var, no config key, not even in tests.
So the optimization is plumbed end-to-end except for the one switch that turns it on, and we pay one redundant write per commit.
Ask
Wire up a public way to enable disable_transaction_file:
- Plumb it through a user-facing surface (e.g.
WriteParams/CommitConfig, with Python and Java bindings kept as thin wrappers), keeping the parameter name consistent across languages. - Add tests covering: a commit with the external file disabled produces a readable dataset from the inline
transaction_section, and the writer feature flag (bit 32) is set.
Note: enabling it sets a writer feature flag, so older readers that don't understand bit 32 will reject the dataset. Defaulting it on is tracked separately (see follow-up issue).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing ManifestWriteConfig in rust/lance/src/dataset.rs into the commit paths in rust/lance/src/io/commit.rs, then inspect the flag definition in rust/lance-table/src/feature_flags.rs and transaction handling in rust/lance-table/src/io/manifest.rs. Identify the user-facing write configuration and its Python and Java wrappers. Done means callers can enable the option, a commit remains readable from the inline transaction_section, and writer feature flag bit 32 is set.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python, rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100