Tracking: Stability and Performance Optimization of Trillion-row Lance Tables

Background & Goals

Targeting tables at the scale of one trillion rows / millions of fragments, this effort profiles Lance's stability and performance across metadata / read / write / index, and drives the corresponding optimizations to land.

This issue serves as the umbrella entry point, tracking the known problems and optimization tasks per module.

Test Dataset

Module 1: Lance-Core (Kernel)

Continuously being expanded.

Metadata

• Oversized manifest / inline-transaction redundancy: the manifest inlines the transaction by default, so under heavy-write workloads the fragment info is stored three times (once in the manifest struct + once in the inline transaction + once in the standalone _transactions/*.txn), ~1.6 GB per version. https://github.com/lance-format/lance/issues/7961
• O(n*m) fragment compares in build_manifest for Update/Delete https://github.com/lance-format/lance/issues/8209

Write

• Session sharing in the Fragment API: sharing currently only exists at the dataset level; Fragment-level operations cannot reuse it, which amplifies overhead when the many small operations. https://github.com/lance-format/lance/issues/7974

Read

• The Java blocking_dataset exposes more metadata information, such as Fragments statistics, to optimize the Read Plan process for engines like Spark. https://github.com/lance-format/lance/issues/8071
• (TBD: profiling and optimization items for point lookups / range scans / projec reads at the hundred-billion scale)

Index

• Adapting zonemap-based partitioning to Compaction: Compaction may break the precondition that "cluster values are unique within a fragment," causing zonemap partition pruning to become ineffective — an adaptation scheme is needed.
• Bloom filter BinaryArray i32 offset overflow: i32 offset overflows on large BinaryArrays. https://github.com/lance-format/lance/issues/7973
• zonemap / bloomfilter index https://github.com/lance-format/lance/issues/8468

Compaction

• Compaction plan performance https://github.com/lance-format/lance/issues/7967
• Compaction skip collect rowids to avoid oom during compaction commit https://github.com/lance-format/lance/pull/7778
• Support max_source_rows and max_source_bytes limits https://github.com/lance-format/lance/issues/8234

Module 2: Lance-Spark (Connector)

• Hot-read on the manifest during Append: the schema-fetch path triggers hot the large manifest.
• Hot-read on the manifest during Update / Add Columns / Compaction: the data mechanism causes repeated reads of the large manifest (DatasetCache vs. Session Share design choice).
• Driver OOM on Compaction commit: aggregating row id maps on the driver causes OOM.
• Write task count is not controllable: RequiresDistributionAndOrdering doe implement requiredNumPartitions(), so write parallelism can only be worked around indirectly via AQE parameters and cannot be specified directly, which easily leads oversized per-task buffering → off-heap OOM.
• The split plan takes more than two minutes under scenarios with large data volumes and zonemap indexes in Lance Spark.

This is an umbrella issue covering Lance-Core metadata, read, write, index and compaction work plus Lance-Spark paths such as DatasetCache, Session Share, compaction commit and split planning. Start by selecting one unchecked item and reading its linked issue when available; done means the selected bottleneck is optimized and validated against the stated large-table scenario.