Feature Request: Use zstd trained dictionaries for miniblock compression

It might be interesting to see if you could use a fixed trainable dictionary for zstd column encoding.

I think that zstd offers the ability to train a dictionary you can then utilize to compress/decompress your dataset with (https://github.com/facebook/zstd?tab=readme-ov-file#the-case-for-small-data-compression).

If enabled, this could increase both compression and efficiency by enabling either dict per file, or dict per dataset.

Here's an AI generated summary of some ideas I looked into:

Summary

Lance compresses miniblock chunks independently at ~4 KiB granularity using zstd, but each chunk starts with zero history. Zstd's trainable dictionary feature is designed exactly for this scenario — many small, structurally similar data items — and could significantly improve compression ratios at no additional I/O cost by following the existing FSST precedent.

Problem

Lance's dominant compression unit is the miniblock chunk, which targets ~4 KiB (hard max ~8 KiB):

• ValueEncoder aims for 4 KiB chunks (e.g., 1024 x Int32, 512 x Float64)
• BinaryMiniBlockEncoder targets 4 KiB via AIM_MINICHUNK_SIZE
• Each chunk is compressed as an independent zstd frame — no shared context across chunks

The current ZstdBufferCompressor reuses the zstd::bulk::Compressor object across chunks within a page, but this only avoids memory re-allocation — it does not carry forward any match history or statistical knowledge between chunks. Each 4 KiB chunk compresses from scratch.

At 4 KiB, zstd has very little data to learn patterns from. This is well within the range (256 bytes – ~8 KB) where zstd's documentation states trained dictionaries provide the most benefit:

Zstd can use dictionaries to improve compression of small data, on both compression speed and compression ratio. The dictionary is trained from a set of samples. [...] Compression gains are mostly effective in the first few KB.

Additionally, GeneralMiniBlockCompressor skips compression entirely when buffers are under MIN_BUFFER_SIZE_FOR_COMPRESSION (4 KiB), and a test comment in block.rs:808 explicitly notes: "Need to use large pages as small pages might be too small to compress". A trained dictionary could make compression viable even at these small sizes.

Proposed solution

Train a zstd dictionary per page, per column, and store it inline in the encoding protobuf — exactly how FSST stores its symbol table today.

Why per-page, inline in the encoding proto

Scope	Extra I/O at read	Storage	Fit
Per-page (inline in proto)	0 — rides with column metadata bulk read	~32–112 KB per page in metadata section	Best — exact FSST precedent
Per-column (column buffer)	1 IOP per column per scan	One dict per column per file	Adds latency, especially on cloud
Per-dataset	1 IOP per column per dataset open	One dict per column, shared across fragments	Fragile; stale on distribution shift; management overhead

The per-page approach has zero additional I/O overhead. The column metadata section is already read in bulk as a mandatory step before any page decode (typically captured in the single tail read). A dictionary stored as bytes in the encoding proto is available the moment the metadata is parsed — identical to how FsstMiniBlockDecompressor loads description.symbol_table today.

Training data is already in memory

When encoding a page, Lance has already accumulated ~8 MiB of raw column data (cache_bytes_per_column default). That's ~2,000 chunks of ~4 KiB — well above zstd's recommendation of "a few thousand samples." The dictionary is trained from a sample of those chunks, then all chunks are compressed with it. No extra I/O on the write path either.

Proto change

Following the FSST precedent (encodings_v2_1.proto:341–356):

message General {
  BufferCompression compression = 1;
  CompressiveEncoding values = 2;
  bytes dictionary = 3;  // optional trained zstd dictionary
}

Write path

In GeneralMiniBlockCompressor::compress:

1. Collect a sample of raw miniblock chunks from the page (already in memory)
2. Call zstd::dict::from_samples() (Rust) / ZDICT_trainFromBuffer() to train a dictionary (~6 MB working memory, fast)
3. Create a zstd::bulk::Compressor with the dictionary via Compressor::with_dictionary()
4. Compress all chunks using the dictionary-aware compressor
5. Store the dictionary bytes in the General.dictionary proto field

Read path

In GeneralMiniBlockDecompressor:

1. On construction, read General.dictionary from the already-parsed encoding proto (zero I/O — it's in the column metadata)
2. Create a zstd::bulk::Decompressor with the dictionary (or use ZSTD_decompress_usingDDict)
3. Decompress chunks as today, but with the dictionary providing initial context

Backward compatibility

• Files without General.dictionary (all existing files) continue to work — the field is optional, and the decompressor falls back to standard zstd
• This could be gated behind a file version check (V2.2+ or a new minor version) and/or a feature flag
• The CompressionParams / field metadata system already supports per-column configuration, so dictionary training could be opt-in initially

Costs and tradeoffs

Dimension	Impact
Read I/O	Zero additional — dictionary is in the metadata bulk read
Read CPU	One ZSTD_createDDict call per page (sub-millisecond; could be cached in LanceCache)
Write CPU	One ZDICT_trainFromBuffer per page (~milliseconds on ~8 MiB of samples)
Metadata size	+32–112 KB per page per column (tunable via maxdict parameter)
Complexity	Modest — follows established FSST pattern; proto change is additive

For a file with 100 pages of one column, a 100 KB dictionary per page adds ~10 MB to the metadata section, against ~800 MB of raw data. This is proportional and likely offset by improved compression of the data section.

Prior art in the codebase

• FSST (encodings/physical/fsst.rs): Trains a symbol table (~2 KB) per page, stores it as bytes symbol_table in the Fsst proto, loads it from the encoding proto at decode time with zero extra I/O. This is the exact pattern proposed here.
• ZstdBufferCompressor (encodings/physical/block.rs): Already caches the Compressor context via OnceLock<Mutex<Compressor>> for memory reuse. Adding dictionary support would extend this to also load the dictionary into the context.
• Rust zstd crate (v0.13, already a dependency): Supports Compressor::with_dictionary(), zstd::dict::from_samples(), and zstd::bulk::Decompressor with dictionary — no new dependencies needed.

Open questions

• What dictionary size gives the best size/speed tradeoff for Lance's typical chunk sizes? (Benchmark needed — 32 KB vs 64 KB vs 112 KB)
• Should dictionary training be opt-in via CompressionParams or automatic when zstd is selected?
• Should the MIN_BUFFER_SIZE_FOR_COMPRESSION threshold (4 KiB) be lowered when a dictionary is available, since dictionaries make small-buffer compression viable?
• Is there value in a per-column (per-file) dictionary for cases where metadata size is a concern? This would add 1 IOP per column but reduce dictionary storage from O(pages) to O(1) per column.

Start with encodings/physical/block.rs, encodings/physical/fsst.rs, and encodings_v2_1.proto to understand existing miniblock compression and inline metadata patterns. Benchmark dictionary sizes and training scope against current zstd behavior, then clarify the configuration, compatibility, and metadata tradeoffs; done requires an agreed design and validated compression results.