Lance compression is ineffective for sparse matrix COO data

Lance datasets storing sparse matrix data in coordinate (COO) format achieve minimal compression compared to other columnar formats like Parquet. This results in storage sizes that are nearly identical to raw bytes, making Lance unsuitable for large-scale sparse data workloads.

Expected Behavior

For sparse matrix data with repetitive integer patterns (cell IDs, gene IDs), we should see significant compression (3-5x) similar to other columnar formats.

Actual Behavior

Lance storage size ≈ raw bytes with minimal compression (~1.1x), regardless of:

Group size settings (max_rows_per_group)
File size settings (max_rows_per_file)
Dataset compaction (dataset.optimize.compact_files())
Chunked vs single-write approaches

Impact

This prevents adoption of Lance for scientific computing workloads involving sparse matrices (e.g., single-cell genomics) where datasets can be 100B+ non-zero entries, resulting in prohibitive storage costs.

Data Characteristics

The test data mimics real-world sparse matrices:

Schema: cell_integer_id: uint32, gene_integer_id: uint16, value: uint16
Density: 2.5% (typical for single-cell data)
Size: 12.5M non-zero entries (~100MB raw) for this test dataset

Note, my "real-world" test is about 8B non-zero entires and takes 62GB to store (vs. the native sparse coo hdf5 file takes 18GB). This is less than 5% of my dataset. Actual dataset in the industry circa 2025 are going to be ~100-200B non-zero entries. Therefore storage is as important as query performance to convince existing users to adopt a new format

Typical result

Raw file/directory sizes:
• test_snappy.parquet: 33,798,684 bytes
• test_zstd.parquet: 31,972,975 bytes
• lance_single.lance/: 100,002,066 bytes
• lance_chunked.lance/: 257,006,229 bytes

Note that sequential data chunks appended to a lance table, followed by post-write optimization doesn't change the fact that chunked table is so much larger.

Environment

Lance version: 0.30.0
PyArrow version: 20.0.0
Python: 3.12
OS: macOS

Minimal reproducible code

#!/usr/bin/env python3
"""
Minimal reproducible example for Lance compression issue.

This script demonstrates that Lance datasets storing sparse matrix data in COO format
achieve minimal compression compared to other columnar formats like Parquet.

Expected behavior: Significant compression due to repetitive integer patterns
Actual behavior: Storage size nearly equals raw bytes (no compression)
"""

import os
import shutil
from pathlib import Path

import lance
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq


def generate_sparse_matrix_coo(n_cells: int, n_genes: int, density: float = 0.025):
    """
    Generate a sparse matrix in COO format similar to single-cell RNA-seq data.

    Args:
        n_cells: Number of cells (rows)
        n_genes: Number of genes (columns)
        density: Fraction of non-zero entries

    Returns:
        Tuple of (cell_ids, gene_ids, values) arrays
    """
    print(
        f"Generating sparse matrix: {n_cells:,} cells × {n_genes:,} genes ({density:.1%} density)"
    )

    # Calculate number of non-zero entries
    n_nonzero = int(n_cells * n_genes * density)
    print(f"Non-zero entries: {n_nonzero:,}")

    # Generate random cell and gene indices
    # Use patterns that should compress well (sorted, repetitive)
    np.random.seed(42)  # For reproducibility

    # Create realistic patterns: some genes are highly expressed, some cells have more expression
    cell_ids = np.random.randint(0, n_cells, size=n_nonzero, dtype=np.uint32)
    gene_ids = np.random.randint(0, n_genes, size=n_nonzero, dtype=np.uint16)

    # Sort by cell_id then gene_id for better compression (realistic access pattern)
    sort_idx = np.lexsort((gene_ids, cell_ids))
    cell_ids = cell_ids[sort_idx]
    gene_ids = gene_ids[sort_idx]

    # Generate expression values (uint16 range: 0-65535)
    # Use realistic distribution with many low values
    values = np.random.exponential(scale=10, size=n_nonzero).astype(np.uint16)
    values = np.minimum(values, 65535)  # Clip to uint16 max

    return cell_ids, gene_ids, values


def create_arrow_table(cell_ids, gene_ids, values):
    """Create PyArrow table with optimized schema."""
    return pa.table(
        {
            "cell_integer_id": pa.array(cell_ids, type=pa.uint32()),
            "gene_integer_id": pa.array(gene_ids, type=pa.uint16()),
            "value": pa.array(values, type=pa.uint16()),
        }
    )


def get_directory_size(path: Path) -> int:
    """
    Get total size of directory in bytes (equivalent to `du -sb`).
    Lance datasets are directories, not single files.
    """
    if path.is_file():
        return path.stat().st_size

    total = 0
    for root, dirs, files in os.walk(path):
        for file in files:
            file_path = Path(root) / file
            try:
                total += file_path.stat().st_size
            except (OSError, FileNotFoundError):
                # Handle race conditions or permission issues
                continue
    return total


def compact_lance_dataset(dataset_path: Path):
    """Compact Lance dataset for optimal storage."""
    print(f"  Compacting {dataset_path.name}...")
    dataset = lance.dataset(str(dataset_path))
    dataset.optimize.compact_files(target_rows_per_fragment=1_000_000)


def test_compression_comparison():
    """Compare compression across formats using the same dataset."""

    # Test parameters - single dataset for fair comparison
    n_cells = 50_000  # 50K cells
    n_genes = 10_000  # 10K genes
    density = 0.025  # 2.5% density
    chunk_size = 10_000  # 10K records per chunk

    # Calculate expected storage
    n_nonzero = int(n_cells * n_genes * density)
    raw_bytes = n_nonzero * (4 + 2 + 2)  # uint32 + uint16 + uint16
    print(f"Expected raw storage: {raw_bytes / 1e9:.2f} GB")

    # Generate test data once
    cell_ids, gene_ids, values = generate_sparse_matrix_coo(n_cells, n_genes, density)
    table = create_arrow_table(cell_ids, gene_ids, values)

    # Setup output directory
    output_dir = Path("../slaf-datasets/scratch")
    output_dir.mkdir(parents=True, exist_ok=True)

    # Clean up previous runs
    for old_file in output_dir.glob("*"):
        if old_file.is_file():
            old_file.unlink()
        elif old_file.is_dir():
            shutil.rmtree(old_file)

    print(f"\nUsing output directory: {output_dir.absolute()}")

    # === Test Parquet Baseline ===
    print("\n=== Testing Parquet ===")

    # Parquet with snappy
    parquet_snappy_path = output_dir / "test_snappy.parquet"
    pq.write_table(
        table,
        parquet_snappy_path,
        compression="snappy",
        row_group_size=1_000_000,
    )
    parquet_snappy_size = parquet_snappy_path.stat().st_size
    parquet_snappy_ratio = raw_bytes / parquet_snappy_size
    print(
        f"Parquet (snappy): {parquet_snappy_size / 1e9:.2f} GB ({parquet_snappy_ratio:.2f}x)"
    )

    # Parquet with zstd
    parquet_zstd_path = output_dir / "test_zstd.parquet"
    pq.write_table(
        table,
        parquet_zstd_path,
        compression="zstd",
        compression_level=3,
        row_group_size=1_000_000,
    )
    parquet_zstd_size = parquet_zstd_path.stat().st_size
    parquet_zstd_ratio = raw_bytes / parquet_zstd_size
    print(
        f"Parquet (zstd): {parquet_zstd_size / 1e9:.2f} GB ({parquet_zstd_ratio:.2f}x)"
    )

    # === Test Lance Single Write ===
    print("\n=== Testing Lance Single Write ===")
    lance_single_path = output_dir / "lance_single.lance"

    lance.write_dataset(
        table,
        str(lance_single_path),
        mode="overwrite",
        max_rows_per_file=10_000_000,
        max_rows_per_group=2_000_000,
        max_bytes_per_file=50 * 1024**3,
    )

    # Compact after writing
    compact_lance_dataset(lance_single_path)

    lance_single_size = get_directory_size(lance_single_path)
    lance_single_ratio = raw_bytes / lance_single_size
    print(
        f"Lance (single write): {lance_single_size / 1e9:.2f} GB ({lance_single_ratio:.2f}x)"
    )

    # === Test Lance Chunked Write ===
    print("\n=== Testing Lance Chunked Write ===")
    lance_chunked_path = output_dir / "lance_chunked.lance"

    # Create initial empty dataset with same schema
    empty_table = pa.table(
        {
            "cell_integer_id": pa.array([], type=pa.uint32()),
            "gene_integer_id": pa.array([], type=pa.uint16()),
            "value": pa.array([], type=pa.uint16()),
        }
    )

    lance.write_dataset(
        empty_table,
        str(lance_chunked_path),
        mode="overwrite",
        max_rows_per_file=10_000_000,
        max_rows_per_group=2_000_000,
        max_bytes_per_file=50 * 1024**3,
    )

    # Write in chunks
    total_records = len(cell_ids)
    n_chunks = (total_records + chunk_size - 1) // chunk_size
    print(f"Writing {total_records:,} records in {n_chunks} chunks of {chunk_size:,}")

    for i in range(n_chunks):
        start_idx = i * chunk_size
        end_idx = min((i + 1) * chunk_size, total_records)

        chunk_table = pa.table(
            {
                "cell_integer_id": pa.array(
                    cell_ids[start_idx:end_idx], type=pa.uint32()
                ),
                "gene_integer_id": pa.array(
                    gene_ids[start_idx:end_idx], type=pa.uint16()
                ),
                "value": pa.array(values[start_idx:end_idx], type=pa.uint16()),
            }
        )

        lance.write_dataset(
            chunk_table,
            str(lance_chunked_path),
            mode="append",
            max_rows_per_file=10_000_000,
            max_rows_per_group=2_000_000,
            max_bytes_per_file=50 * 1024**3,
        )

        if (i + 1) % 10 == 0 or (i + 1) == n_chunks:
            print(f"  Wrote chunk {i+1}/{n_chunks}")

    # Compact after writing (same as single write)
    compact_lance_dataset(lance_chunked_path)

    lance_chunked_size = get_directory_size(lance_chunked_path)
    lance_chunked_ratio = raw_bytes / lance_chunked_size
    print(
        f"Lance (chunked write): {lance_chunked_size / 1e9:.2f} GB ({lance_chunked_ratio:.2f}x)"
    )

    # === Summary ===
    print("\n" + "=" * 60)
    print("COMPRESSION COMPARISON SUMMARY")
    print("=" * 60)
    print(f"Raw bytes: {raw_bytes / 1e9:.2f} GB")
    print(
        f"Parquet (snappy): {parquet_snappy_size / 1e9:.2f} GB ({parquet_snappy_ratio:.2f}x)"
    )
    print(
        f"Parquet (zstd):   {parquet_zstd_size / 1e9:.2f} GB ({parquet_zstd_ratio:.2f}x)"
    )
    print(
        f"Lance (single):   {lance_single_size / 1e9:.2f} GB ({lance_single_ratio:.2f}x)"
    )
    print(
        f"Lance (chunked):  {lance_chunked_size / 1e9:.2f} GB ({lance_chunked_ratio:.2f}x)"
    )

    # Calculate differences
    chunked_vs_single_pct = (
        (lance_chunked_size - lance_single_size) / lance_single_size
    ) * 100
    lance_vs_parquet_gap = parquet_zstd_ratio / lance_single_ratio

    print(f"\nKey Findings:")
    print(f"• Chunked vs Single Lance: {chunked_vs_single_pct:+.1f}% size difference")
    print(f"• Lance vs Parquet gap: {lance_vs_parquet_gap:.1f}x worse compression")
    print(f"• Best compression: Parquet (zstd) at {parquet_zstd_ratio:.1f}x")
    print(f"• Lance compression: {lance_single_ratio:.1f}x (essentially uncompressed)")

    print(f"\nFiles saved to: {output_dir.absolute()}")
    print("\nRaw file/directory sizes:")
    print(f"• {parquet_snappy_path.name}: {parquet_snappy_size:,} bytes")
    print(f"• {parquet_zstd_path.name}: {parquet_zstd_size:,} bytes")
    print(f"• {lance_single_path.name}/: {lance_single_size:,} bytes")
    print(f"• {lance_chunked_path.name}/: {lance_chunked_size:,} bytes")

    return {
        "raw_bytes": raw_bytes,
        "parquet_snappy_size": parquet_snappy_size,
        "parquet_zstd_size": parquet_zstd_size,
        "lance_single_size": lance_single_size,
        "lance_chunked_size": lance_chunked_size,
        "parquet_snappy_ratio": parquet_snappy_ratio,
        "parquet_zstd_ratio": parquet_zstd_ratio,
        "lance_single_ratio": lance_single_ratio,
        "lance_chunked_ratio": lance_chunked_ratio,
    }


if __name__ == "__main__":
    print("Lance Compression Reproduction Script")
    print("=====================================")

    # Run simplified compression comparison
    results = test_compression_comparison()

    print(f"\n{'='*60}")
    print("GITHUB ISSUE SUMMARY")
    print(f"{'='*60}")
    print("Issue: Lance achieves minimal compression on sparse COO data")
    print(f"Lance (single): {results['lance_single_ratio']:.2f}x compression")
    print(f"Lance (chunked): {results['lance_chunked_ratio']:.2f}x compression")
    print(f"Parquet (zstd): {results['parquet_zstd_ratio']:.2f}x compression")
    print(f"Expected: >3x compression due to repetitive integer patterns")
    print(
        f"Gap: {results['parquet_zstd_ratio'] / results['lance_single_ratio']:.1f}x worse than Parquet"
    )

Questions

• What can I expect from Lance v2.0 compression relative to Parquet's codec options?
• For Python users, Are there Lance-specific compression settings not exposed in write_dataset()?
• Is this a known limitation for repetitive integer data?
• Roadmap: Are there plans to improve compression for this use case in v2.1?
• Do you recommend falling back to a version preceding v2.0 until v2.1 is ready?

Workarounds Attempted

✅ Optimized data types (uint32/uint16 instead of int32/float32)
✅ Large "row groups" (10M rows per group)
✅ Dataset compaction after writing
✅ Sorted data for locality
❌ None achieved significant compression

Request

I'd appreciate:

• Confirmation if this is expected behavior
• Guidance on enabling compression if available
• Roadmap for compression improvements
• Alternative approaches for sparse data in Lance

This issue blocks adoption of Lance for large-scale scientific computing where storage efficiency is critical.

Broader context

I'm building a cloud-native storage format on top of lance, and single-node compute backend for single cell data that takes the best ideas from zarr, dask, duckdb and lancedb: https://slaf-project.github.io/slaf

Run the provided Python reproducer with Lance 0.30.0 and compare the reported Parquet and Lance sizes for single and chunked writes. Investigate the Lance storage and compression path responsible for these datasets, then verify that the same sparse COO workload achieves materially improved compression without breaking the existing write and compaction behavior.