feat: Split version manifests into a hierarchical, independently loadable fragment catalog

Summary

Lance currently stores the complete fragment list inline in each version manifest. This is simple and works well at modest scale, but it makes manifest I/O, protobuf decoding, memory use, and commit serialization grow with the total number of fragments in the snapshot.

At a sufficiently large fragment count, the manifest is no longer merely metadata; it is a small dataset that every reader and writer is still required to process as one protobuf.

I propose evolving the layout toward an Iceberg-style hierarchy:

_versions/<version>.manifest
    ├── schema, config, version metadata, feature flags, ...
    └── fragment manifest descriptors
            ├── manifest/<uuid-1>.frags
            ├── manifest/<uuid-2>.frags
            └── manifest/<uuid-N>.frags
                    └── bounded subsets of Fragment metadata

The version manifest would remain the atomic snapshot root, but it would contain only compact descriptors for independently addressable fragment-manifest files. Descriptors would carry enough summary metadata to prune irrelevant files before fetching them. Selected files could then be loaded concurrently with bounded I/O parallelism.

This is intended to borrow the useful layering from Iceberg, not to copy Iceberg's format or semantics wholesale.

Motivation

Today, opening a version whose manifest is larger than the tail prefetch requires fetching the remaining manifest bytes and decoding the full pb::Manifest. Because fragments is a repeated field in that message, opening the dataset eagerly materializes the complete fragment collection.

The flat layout has several scaling consequences:

• Read amplification: even a query that ultimately needs a small subset of fragments must first download and decode all fragment metadata.
• Limited metadata pruning: there is no independently readable upper level with fragment-ID ranges, or other summaries that can eliminate groups of fragments before their detailed metadata is fetched.
• Limited network concurrency: CPU-parallel protobuf decoding can help after bytes arrive, but it cannot parallelize object-store reads while the fragment catalog is one opaque object.
• Write amplification: a new version serializes the complete current fragment list, even when a commit adds or changes only a small subset.
• Memory and cache amplification: the full fragment vector becomes part of every fully loaded manifest. Immutable fragment-manifest files would instead be cacheable and reusable across versions.
• Higher sensitivity to storage latency: large metadata reads are especially visible on object stores and remote filesystems, and are multiplied by short-lived or distributed workers that repeatedly open the dataset.

There are already related community reports and discussions:

• Discussion #4000: Scale Fragments beyond a flat list
• Issue #5194: Lance commits are progressively slower once a lot of fragments are written
• Issue #4982: Support an API to return parts of a manifest
• Issue #7024: Slow dataset load on an HDFS-backed dataset with 2,000 fragments

I do not assume that the manifest layout is the sole cause of every latency reported in those issues. Version resolution, conflict handling, connection setup, and repeated opens can also contribute. However, the current layout imposes an unavoidable O(total fragments) metadata path on both opens and commits, so it is worth isolating and benchmarking directly.

Proposed layout

1. Keep the version manifest as the atomic root

The version manifest should continue to hold snapshot-wide state such as:

• version and writer information;
• schema and table configuration;
• feature flags;
• transaction and index-section references;
• a compact ordered list of fragment-manifest descriptors.

Committing the root remains the atomic visibility point. Readers never observe a partially committed snapshot because sidecar files are immutable and written before the root references them.

2. Store fragment metadata in bounded, immutable files

Each file under manifest/ would contain a subset of the version's fragment metadata. Files should have a target encoded size or target fragment count so that no single fragment-manifest file becomes another unbounded metadata monolith.

An illustrative descriptor is:

message FragmentManifestFile {
  string path = 1;
  uint64 size_bytes = 2;
  uint32 num_fragments = 3;

  optional uint64 min_fragment_id = 4;
  optional uint64 max_fragment_id = 5;
  optional uint64 num_rows = 6;
  optional uint64 data_size_bytes = 7;
}

The exact fields are open for discussion. The important property is that a reader can decide whether a fragment-manifest file may be relevant without first reading all of its fragment entries.

Useful summaries could include:

• fragment-ID range, and optionally a compact coverage representation when ranges are sparse;
• fragment, row, data-file, and deletion counts;
• selected metadata statistics that are stable, cheap to maintain, and demonstrably useful for planning.

3. Make each fragment-manifest file self-describing

A fragment-manifest file can begin with a compact offset table followed by contiguous encoded fragment entries:

+------------------------+
| fragment offset header |
+------------------------+
| fragment entry 0       |
| fragment entry 1       |
| ...                    |
| fragment entry N       |
+------------------------+

The header allows the reader to locate entry or entry-group byte ranges without sequentially parsing all preceding entries. Because each file is bounded, the offset metadata is bounded as well.

The reader can choose adaptively between:

• one whole-object read for a small file; or
• several coalesced range reads for a larger file.

This decision should consider object size, store capabilities, expected latency, and configured I/O parallelism. More requests are not automatically faster, especially on local filesystems or high-overhead object stores.

Read path

A scan would use the hierarchy as follows:

1. Read the small version manifest.
2. Evaluate the scan's fragment selection or predicate against descriptor summaries.
3. Drop fragment-manifest files that cannot contribute.
4. Fetch the remaining files concurrently with bounded parallelism.
5. Decode entries in parallel where beneficial.
6. Preserve the current fragment ordering and correctness semantics when results are combined.

Examples of future pruning opportunities include:

• an index or row-address plan selects a known fragment-ID range;
• metadata-only operations can answer from descriptor counts without hydrating every fragment;
• a distributed coordinator sends workers only the fragment metadata they need.

Operations that genuinely require the complete fragment set can still load every fragment-manifest file, but they gain bounded parallel fetch and per-file caching.

Write path and reuse across versions

Fragment-manifest files should be immutable and referenced copy-on-write:

• Append: write one or more files for newly added fragments and reuse existing files.
• Delete/update: rewrite only files containing affected fragment entries and reuse the rest.
• Compaction: replace the affected fragment groups while leaving unrelated groups untouched.
• Metadata maintenance: merge undersized files or split oversized files in a manifest-compaction step.

This changes the steady-state metadata rewrite cost from O(total fragments) toward O(changed fragments + number of fragment-manifest descriptors).

The descriptor list itself must not grow without control. A target file size plus manifest compaction is the simplest initial policy. Grouping by contiguous fragment ID is easy to reason about; grouping by data locality may improve pruning but risks skew. A self-balancing tree or Bε-tree, as mentioned in Discussion #4000, remains an interesting longer-term representation if a flat descriptor list proves insufficient.

Existing prototype

I have implemented a working first-stage prototype in a downstream branch. It deliberately keeps the change incremental:

• large fragment lists are moved from the inline manifest protobuf into a sidecar under manifest/;
• the root uses repeated fragment_manifests, even though the current prototype writes only one file, so one-to-many splitting does not require another root format change;
• each sidecar has a self-describing fragment offset table;
• the reader uses object-store I/O parallelism to issue concurrent, coalesced range reads and then decodes fragments in parallel;
• the in-memory Manifest.fragments remains populated after commit, while only the serialized form omits inline fragments;
• new readers transparently support both legacy inline fragments and externalized fragments;
• a reader feature flag makes pre-upgrade readers reject the new layout instead of silently interpreting the inline fragment list as empty;
• end-to-end tests cover commit, reopen, round-trip ordering, stale externalization state, and compatibility behavior.

This prototype already separates fragment bytes from the version root and enables real network-parallel fetch, rather than only CPU-parallel decode after downloading the entire version manifest.

It is intentionally not the complete hierarchical design yet:

• it writes one sidecar containing the full fragment list for each version;
• opening a dataset still eagerly hydrates all fragments to preserve existing APIs;
• it does not yet add summary-based pruning;
• it does not yet reuse unchanged fragment-manifest files across versions;
• multi-file scheduling and manifest compaction remain to be implemented.

I believe this makes it a useful first PR: it establishes the format plumbing, compatibility contract, and independently addressable file layout without requiring an immediate rewrite of every API that currently expects Manifest.fragments to be materialized.

Advantages of this approach

Better scan planning

Iceberg uses a manifest list as an index over manifest files: partition summaries first prune manifests, and only then are individual manifests read. Lance can use the same principle with Lance-specific summaries and fragment semantics.

See:

• Apache Iceberg performance documentation: metadata filtering
• Apache Iceberg specification: manifests and manifest lists

Real I/O parallelism

Independent files and discoverable offsets give the reader multiple fetch units. This creates room for bounded concurrent loading at the object-store layer, rather than relying only on parallel deserialization of a buffer that was fetched serially.

Lower commit amplification

Immutable files can be shared by multiple versions. Small appends and localized rewrites no longer need to reproduce metadata for every unchanged fragment.

More effective caching

Fragment-manifest files are immutable and independently keyed, so unchanged metadata can remain cached across version opens. A single large version-specific protobuf has much less cache reuse.

A natural partial-manifest API

The same descriptors can support local object-store reads, a future Lance Namespace service, or a gRPC metadata service. The storage format does not need to assume that all metadata is transferred as one protobuf.

Incremental adoption

The first stage can preserve the existing in-memory model and API behavior. Lazy fragment materialization and scan-specific loading can follow once the on-disk hierarchy is available and benchmarked.

Costs and risks

This is not a free optimization:

• More metadata objects: small tables should keep inline fragments, and sidecar creation should be threshold-based.
• Small-file growth and skew: writers need target sizes plus split/merge policies.
• More complex cleanup: vacuum must retain sidecars referenced by any retained version and remove only unreachable files.
• Commit orphans: failed commit attempts may leave immutable, unreferenced sidecars that later cleanup must collect.
• More complicated conflict resolution: retries must not accidentally publish stale sidecar references.
• Potentially excessive range requests: readers need a whole-file-versus-range-read heuristic and bounded concurrency.
• API assumptions: many code paths currently expect the complete Arc<Vec<Fragment>>; introducing true laziness will require a staged refactor.

These costs are manageable, but they should be part of the design rather than deferred until after the format is introduced.

Compatibility and correctness

I suggest the following contract:

• New readers continue to read legacy manifests with inline fragments.
• Externalized manifests set a reader feature flag.
• Old readers fail with an unsupported-feature error instead of observing an empty dataset.
• Fragment-manifest files are immutable after publication.
• The root version manifest is committed only after every referenced file is durable.
• Fragment ordering is explicit and deterministic across files.
• Counts, ranges, file sizes, and offset tables are validated while reading.
• Cleanup traces references from every retained version, including branches, tags, detached versions, and shallow clones where applicable.

The format should not depend on directory listing for correctness; all live files are reached from the committed root.

Suggested rollout

1. Benchmark the flat layout

   • 1K, 10K, 100K, and 1M fragments;
   • local filesystem, a cloud object store, and a remote filesystem such as HDFS;
   • open latency, commit latency, bytes fetched, request count, decode CPU, and peak memory;
   • full-scan planning, selective scans, small appends, updates, and compaction.

2. Land the one-sidecar compatibility layer

   • externalize above a threshold;
   • dual-read old and new layouts;
   • reader feature flag;
   • adaptive whole-file/range reads;
   • format and end-to-end tests.

3. Split into bounded multiple files

   • concurrent inter-file loading;
   • deterministic grouping;
   • size limits and validation.

4. Add copy-on-write reuse and manifest compaction

   • reuse unchanged files across versions;
   • merge small files and split large or skewed files;
   • integrate reference-aware cleanup.

5. Add summaries and lazy planning

   • descriptor-level fragment and data pruning;
   • metadata-only operations;
   • avoid hydrating all fragments for selective scans and distributed workers.

The benchmark should compare the current flat layout, the one-sidecar prototype, and the full multi-file hierarchy. That will show which improvements come from smaller roots, network concurrency, pruning, caching, or copy-on-write reuse.

Open questions

1. Should the first grouping policy use encoded byte size, fragment count, contiguous fragment IDs, data locality, or a combination?
2. Which descriptor summaries are valuable enough to maintain in the first version?
3. Should the descriptor list stay inline in the version manifest, or have its own external level once it crosses a threshold?
4. How much of Manifest.fragments can become lazy without destabilizing existing Rust, Python, and Java APIs?
5. What is the correct whole-file-versus-range-read heuristic for local, cloud, and remote filesystem backends?
6. How should fragment-manifest compaction interact with normal data compaction?
7. Should the longer-term structure remain an Iceberg-like bounded list, or evolve into a balanced tree if skew or descriptor-list growth becomes material?

Request for feedback

Would the maintainers be open to an incremental PR starting with the one-sidecar format and compatibility plumbing described above, followed by benchmarks and the multi-file/pruning work?

I am happy to upstream the prototype, add the requested benchmark matrix, and adjust the on-disk schema based on feedback before treating the layout as stable.

Start by locating the version-manifest protobuf (pb::Manifest) and the in-memory Manifest.fragments handling described in the issue. Review existing commit, reopen, round-trip ordering, stale-externalization, and compatibility tests, then define the sidecar format and reader/writer changes needed for the prototype stage. Done means externalized fragments remain compatible with legacy layouts and the end-to-end tests pass.