[RFC] Lance Table Format Integration: OpenSearch as a Distributed Search Layer for Lance Tables

Is your feature request related to a problem? Please describe

This RFC proposes integrating the Lance table format into OpenSearch. For OpenSearch users, this enables vector, hybrid, and multimodal search, filtering, and aggregations over billion scale corpora served directly from object storage, re-embedding without reindexing, and search over lakehouse data without copying it into the cluster. Lance suits this workload where analytics oriented formats do not. Its file layout serves a single row fetch in 1 to 2 IOPs, which is what document retrieval and top-k result fetching demand, and the same columnar layout feeds aggregation scans. For the Lance ecosystem, it supplies the distributed search layer that no open source implementation provides today. An index of this type is backed by a spec compliant Lance table. Data pipelines such as Spark and Ray write the table directly, analytics engines such as DuckDB query it in place, and OpenSearch serves distributed search over it, building search indices that are themselves spec compliant table objects. Accepting document writes through the standard APIs is a planned second stage, outlined in Future work.

Context

Lance

Lance is an open source columnar format designed for multimodal AI workloads. It is a general purpose table format rather than a vector store. One table holds vectors, text, scalars, and blob assets, serves both training and search access patterns, and defines search indices as part of the format. It is a stack of loosely coupled specifications.

Layer	Role
File format	Columnar pages optimized for random access (1 to 2 IOPs per row fetch)
Table format	Manifest based MVCC, fragments, deletion files, zero copy column evolution
Index format	Vector (IVF_PQ, HNSW, RaBitQ), scalar (BTree, Bitmap, Zone Map, Bloom, N-gram, R-Tree), and FTS indices as first class table objects
Catalog / Namespace	Table discovery over existing catalogs (Glue, Unity, Polaris, Iceberg REST)

Motivation

Search is disconnected from the open table ecosystem

The lakehouse model, where one copy of data in an open table format is shared by many engines, is now a well established architecture for analytics. Lance brings the same model to AI data. OpenSearch stands outside this ecosystem. Making lakehouse data searchable requires copying it into the cluster and keeping the copy synchronized, and the search cluster's data is in turn invisible to the engines around it.

Connecting OpenSearch to the table ecosystem removes that boundary. The same table that Spark transforms, Ray embeds, and Trino or DuckDB aggregate becomes directly searchable, joining the aggregation centric world of the lakehouse engines with the search centric world of OpenSearch, without copy pipelines between them. Beyond the connection itself, the format resolves several standing frictions.

1. Re-embedding without reindexing. Changing the embedding model today requires a full reindex, including text structures. With Lance, a new embedding column is added and backfilled without rewriting existing data files, and only its vector index is built. The inverted index is untouched.
2. Multimodal payloads as first class fields. Images, video, and audio live in blob encoded columns of the same table, instead of a separately managed object store with reference bookkeeping.
3. Lower cost at scale. Data and all indices live on object storage, and nodes only cache what they serve. An idle or cold index costs storage alone, and moving an index between hot and warm tiers changes cache policy rather than copying data.

Lance provides these as an open, spec governed format rather than a proprietary storage engine, so adopting it does not couple OpenSearch to a single vendor's implementation.

Lance has no open source distributed search layer

Lance is searchable in a single process today. LanceDB and the Python bindings run vector, full text, and hybrid search embedded in one process. What no open source component provides is distributing one search query across nodes. The Spark integration exposes search functions but plans them as a single input partition invoking the single process API, lance-ray distributes index building and compaction rather than queries, and the Trino connector covers scans and DML with no search path. Distributed search execution exists only commercially, in LanceDB Enterprise (reference). OpenSearch already provides exactly this layer, and this proposal applies it to the Lance table directly, with no copy.

Describe the solution you'd like

This RFC proposes integration at the Lance table format level. An OpenSearch index of this type maps to a Lance base table on object storage, written by external engines such as Spark, Ray, and Flink. OpenSearch participates in the table as two roles.

1. Distributed query layer. Coordinators fan out vector, hybrid, and analytical queries across the index's shards, each a stateless partition of the table's fragments and index segments, and merge partial results.
2. Index builder. OpenSearch builds missing search indices, native Lance vector, scalar, and FTS index types, over rows written by any engine, and commits them back to the table.

The table remains a spec compliant Lance table throughout. Index commits use Lance's optimistic concurrency control, the same mechanism external writers use, so no coordination service is introduced.

        External writers (Spark / Ray / Flink)
                  |  direct commits (Lance OCC)
                  v
     +-------------------------------------+
     |   Lance base table (object storage) |  <- spec compliant, readable and
     |   fragments / manifest /            |     writable by any Lance engine
     |   _indices (vector, scalar,         |
     |   FTS)                              |
     +-------------------------------------+
            ^                  |
            | CreateIndex      | range reads on cache miss
            | commits          v
     +---------------------------------------------+
     | OpenSearch                                  |
     |                                             |
     | Query path: coordinator -> shards           |
     | (stateless fragment partitions across       |
     | data nodes) -> merge -> client              |
     |                                             |
     | Index builder: builds vector, scalar, and   |
     | FTS indices over uncovered fragments        |
     +---------------------------------------------+

Architecture

Lance table fundamentals

This section covers the Lance concepts needed to follow the rest of the design. Authoritative details are in the table format and index format specifications.

A Lance table is a directory of files on object storage or a filesystem.

Manifests and commits. The state of the table at a version is described by a single immutable manifest file, which records the schema, the list of fragments, and the index metadata. Every write operation commits a new manifest. Concurrent writers coordinate through optimistic concurrency, committing with a conditional put on object storage and rebasing or retrying on conflict according to per operation compatibility rules.

Fragments and data files: a two dimensional layout. The table is organized in two dimensions. Rows are grouped into fragments, the horizontal unit. Within a fragment, columns are held by data files, each covering a subset of the columns, the vertical unit. The manifest records how the files of each fragment compose into the full schema.

              columns a..c     column d (added later)
fragment 0    data_file_0x     data_file_0y
fragment 1    data_file_1x     data_file_1y
fragment 2    data_file_2x     data_file_2y

New rows extend the table with new fragments. New columns extend it with one new data file per fragment, and existing files are never rewritten, which is what makes schema evolution cheap. Row deletes are recorded in per fragment deletion files.

Row addresses. A row is identified by a 64 bit row address composed of the fragment id and the row offset within it. Indices reference rows by address. Tables can optionally declare an unenforced primary key, which gives rows a stable identity.

Indices as table objects. Indices live under _indices/ and are registered in the manifest, each segment recording the set of fragments it covers in a bitmap. Coverage is allowed to be partial. Fragments not covered by an index are answered by scanning, so freshly appended data is queryable before it is indexed. Index metadata carries a type identifier, and an engine that does not recognize a type skips those segments and falls back to scanning, which keeps tables readable across engines with different capabilities.

Runtime overview

A cluster serving a Lance backed index runs search and index building over the table, with the following anatomy at runtime.

                          Client
                            |
                     +-------------+
                     | Coordinator |  fan out, merge
                     +-------------+
                            |
                            v
              +----------------------------+
              | Shards on data nodes       |
              | each a stateless partition |
              | of fragments and index     |
              | segments, served from      |
              | local caches               |
              +----------------------------+
                            |  range reads on miss
                            v
   +---------------------------------------------------+
   | Object storage: Lance base table                  |
   | data files, manifests, _indices                   |
   | (vector, scalar, FTS)                             |
   +---------------------------------------------------+
            ^                            ^
            | Append / AddColumns        | CreateIndex commits
            | (external writers)         | (OpenSearch index builder)

A search fans out from the coordinator to the index's shards. A shard of this index type is a stateless search partition, a subset of the table's fragments and the index segments covering them, cached on local disk and memory and faulted in from object storage on miss, the file cache model that searchable snapshots and the warm tier run today. Cached objects never go stale, because data files, deletion files, and index segments are immutable and a version change swaps the file set rather than file contents. A shard presents its fragments through the Lucene reader API, so the query DSL and aggregations execute over it, and it routes columnar operations to the DataFusion runtime that the analytics engine work embeds, which also serves PPL and SQL. The coordinator merges partial results.

Writes reach the table as direct Lance commits by external engines. OpenSearch discovers them by checking out newer manifest versions on a configurable cadence, so search visibility of external writes is the checkout interval. Index builder tasks run on data nodes and commit vector, scalar, and FTS indices over uncovered fragments through the same optimistic concurrency that external writers use.

No node holds authoritative state. The table on object storage is the single source of truth, and cluster state carries only the mapping, the checked out manifest version, and the fragment partition across shards. The subsections below develop each of these pieces.

Shards as stateless search partitions

A shard of this index type owns no authoritative data. The table owns the rows, durability belongs to the object store, and the shard is a deterministic partition of the table's fragments at the checked out version, recomputed at each checkout.

The shard machinery is reused for the jobs that remain necessary, work assignment, replication, failover, and fan out with merge, in line with reader and writer separation. Assumptions from local storage (hash routing, fixed shard count) do not carry over. At the reader API level each leaf is a fragment group at the pinned version, so per segment execution applies unchanged (see Query execution).

The shard also disappears from the user model, in the way query engines such as Spark and Trino derive partitioning from table layout rather than from user configuration. The shard count is derived from the table at attach time, targeting a fixed amount of data per shard; an explicit number_of_shards setting pins the count. Runtime resizing to track growth is deferred (see Future work). Get by _id resolves through the primary key's scalar index.

Query execution

A query fans out from the coordinator to shards, and the coordinator merges partial results. Inside a shard there are two execution routes.

Planned execution runs on the analytics engine. Calcite plans are broken into a DAG of fragments with per backend plan alternatives, columnar fragments execute on the DataFusion backend behind JNI (analytics-backend-datafusion), and Lance plugs into that backend as a data source through its DataFusion TableProvider (lance-datafusion). PPL, SQL, and the DSL front end reach these indices through this route, and filter predicates push down to Lance scalar and FTS indices inside it.

Everything else runs through the Lucene reader API. The shard presents its fragment subset as a DirectoryReader whose leaves are backed by Lance.

1. A leaf corresponds to a fragment group at the pinned manifest version. Docids are row offsets within the group, stable for the reader's lifetime.
2. liveDocs is served from Lance deletion files.
3. DocValues and stored fields are adapters over Lance columns.
4. Text and k-NN queries rewrite to Lance FTS and vector index queries, the way the k-NN plugin wraps native engines today. Term and range predicates execute over doc values or push down to Lance scalar indices.

A query executes on the analytics route when the planner covers it, and on the reader route otherwise.

This split exists for two reasons. The aggregation framework, sorting, scripting, and document and field level security are implemented against readers, so the reader route keeps that feature surface working from day one. The analytics route reuses the planner and runtime already in the sandbox rather than introducing parallel machinery, and the fallback lets its coverage grow without a correctness gap.

Freshness: following external writes

The table is written by external engines through Lance commits, and OpenSearch follows by checking out newer manifest versions on a configurable cadence. Checkout is the visibility boundary, playing the role refresh plays today. New rows become searchable at checkout. Fragments not yet covered by an index are answered by scanning, the behavior the Lance index specification defines for unindexed data, and whether a new version waits for asynchronous index coverage or is exposed immediately with scan evaluation is a per index policy.

Column addition carries the re-embedding story. Adding a column writes new data files per fragment without changing row count, order, or identity, so nothing OpenSearch built is disturbed. Re-embedding a corpus with a new model becomes an external backfill plus an index build, with no reindex of text structures.

Building search structures

A Lance table may already carry the indices search needs, vector, scalar, and FTS, built by external tools such as lance-ray, and OpenSearch uses them as is. The index builder builds the missing ones as ordinary CreateIndex transactions, producing native Lance index types any engine can use.

Text fields support two analysis modes, selected per field in the mapping. With a Lance tokenizer (tokenizer), the FTS index is built directly over the source column, and Lance tokenizes text at build time and query time; tokenizers include lindera and jieba for CJK. With an OpenSearch analyzer (analyzer), the index builder runs that analyzer over the column, writes the resulting tokens to a derived column through an AddColumns commit and backfill, and builds the FTS index over the derived column with whitespace tokenization; queries are run through the same analyzer before being rewritten to FTS queries, so a match query hits exactly as it does on a classic index. The first mode adds nothing to the table beyond the index. The second costs one derived column.

Index builds commit through the same optimistic concurrency external writers use. CreateIndex is compatible with concurrent Append, Delete, Update, and Merge and conflicts only with another CreateIndex on the same index name (verified against the Lance conflict resolver), so builds never block external writers. Derived column backfills commit as Merge operations, which rebase and retry over concurrent appends without blocking them (also verified). Builds are batched per checkout, and one task owns one index name at a time, which avoids the only conflicting case.

Placement is a per index policy. in_table, the default with write access, commits the structure to the table, built once and shared by all nodes. node_local builds it per node as a derived cache, rebuilt on checkout, for read only attachments or to keep commit traffic off the table. A read only table that wants durable shared indices can hold them in a shallow clone referencing the source data through base_paths.

Mapping interface and field type mapping

The mapping is the user facing interface to the table. The design principle is that a mapping is a projection of the Lance schema plus a per field index policy. Each field carries its storage type, which mirrors the Lance column type, and the search structures built over it, chosen among the Lance index types.

The mapping is derived from the Lance schema and re-derived at every checkout, so the table remains the single source of truth and the mapping cannot drift from it. Defaults are deterministic from table metadata. A string column carrying an FTS index maps to text with its tokenizer taken from the index metadata, and to keyword otherwise. A fixed size list of floats maps to knn_vector with the dimension from the schema and the method from an existing vector index. Attach time configuration is an optional set of override rules, persisted and reapplied on each re-derivation, so columns added later by external writers become searchable without any OpenSearch side change.

{
  "properties": {
    "title":     { "type": "text", "analyzer": "kuromoji" },
    "body":      { "type": "text", "tokenizer": "lindera" },
    "category":  { "type": "keyword", "index_structure": "btree" },
    "embedding": { "type": "knn_vector", "dimension": 960,
                   "method": { "engine": "lance", "name": "ivf_pq" } },
    "image":     { "type": "blob" }
  }
}

Default assignments follow this split.

Mapping type	Storage	Search structure
text	Lance column (source of truth)	Lance FTS index (inverted, BM25)
keyword, numeric, date, boolean, ip	Lance column	Lance scalar index (BTree / Bitmap)
wildcard	Lance column	Lance N-gram index
Tag arrays	Lance list column	Lance Label List index
knn_vector	Lance fixed size list column	Lance vector index (IVF_PQ, IVF_HNSW, RaBitQ)
geo_*	Lance column	Lance R-Tree (future)
object / nested	Lance struct / list of struct	per subfield policy
_source	Lance columns (per field random access)	not applicable
blob (new type)	Lance blob column	not applicable, fetched by reference
other Arrow types (binary, decimal, map)	Lance column	none, stored only, retrievable and scan filterable

Types whose semantics are a Lucene data structure rather than a value, such as join, percolator, rank_features, and completion, are out of scope. Workloads that need them stay on classic indices.

Beyond per field structures, Zone Maps and Bloom Filters serve fragment level pruning for scans and aggregations, and aggregations themselves execute over columnar scans on the analytics route (see Query execution). Time travel surfaces as an extension of point in time semantics, where a PIT pins a manifest version, and snapshot and restore reduce to tagging a version and checking it out.

This interface has consequences that improve on the classic mapping model.

1. Multi fields without storage duplication. A text field with a keyword subfield today stores the value in the inverted index and again in doc values. Here it is one Lance column carrying two search structures, an FTS index and a BTree.
2. Field renames become metadata operations. Lance identifies fields by immutable integer IDs, and the mapping binds names to IDs. Renaming a field rewrites no data, which the classic mapping cannot offer at all.
3. Analysis configuration travels with the table. Tokenizer settings live in the FTS index's metadata, so an attached table reproduces its analysis behavior on any cluster.
4. Dynamic mapping follows the table. New columns added by external engines appear in the mapping after checkout.

Serving _source from Lance columns gives per field fetch at 1 to 2 IOPs and removes the stored fields duplication, in the same spirit as derived source but with random access columnar reads.

Attaching a table

The primary path registers a Lance Namespace catalog once. Tables in the namespace surface as indexes automatically, in the way query engines expose catalog tables without per table registration. Attaching a single table by root URI remains available for tables outside any catalog. In both cases OpenSearch derives the mapping from the schema, derives the shard count from the table size, partitions the fragments across shards, and serves queries, using the indices the table already carries and building missing ones (see Building search structures). No data moves, no reindex precedes the first query, and no per table configuration is required.

Preconditions apply. The Lance schema must map to a mapping, following the derivation defaults, with optional override rules where the defaults resolve a column differently than intended. Get by _id requires the table to declare an unenforced primary key, since only the writer knows which column identifies rows. A declared key without a scalar index still serves gets through filtered scans until the index builder creates one (see Building search structures). Without a declaration the index serves search, aggregations, and vector queries, and only the _id APIs are absent. The table's reader feature flags must be understood by the bundled Lance core. Tables laid out for training workloads (very large fragments, blob heavy) work but may warrant a search oriented recompaction.

Relationship to existing work

#20644 pluggable component bundling. #20644 proposes pluggable component bundling so that formats other than Lucene coexist within OpenSearch, defining DataFormat, IndexingExecutionEngine, Committer, CatalogSnapshot, MergeHandler, and Searcher as extension points and naming Lance as a candidate format. Those extension points target composite indices, where a Lucene inverted index and a non Lucene columnar format coexist inside one shard. Lance carries its own search indices, so no Lucene runs alongside it, and those abstractions reduce to identity wrappers with no cross format work to do. The standalone plugin form follows #19653's guidance that engine implementations live in separate plugin packages, with opensearch-jvector as a direct precedent. Lance's own format versioning (feature flags and file format versions) still meets the #20644 goal of decoupling format versions from OpenSearch versions.

Analytics engine and SQL plugin. The sandbox analytics-engine (#21403) provides a DataFusion backend for querying non Lucene indices, and sql#5246 (on the feature/mustang-ppl-integration branch) adds PPL and SQL access. Lance plugs into that framework rather than paralleling it, entering the DataFusion backend as a data source and reusing the Calcite planner (see Query execution).

Parquet data format and composite engine. The sandbox parquet-data-format plugin uses the composite engine to pair Parquet columns with Lucene, because Parquet has no search indices of its own. Lance is columnar as well but carries its own search indices, so it does not need the pairing and takes the standalone form. Both shapes stay available in OpenSearch.

#22594 Arrow format as pull-based ingestion source. That RFC brings Arrow sources, including Lance, into standard Lucene backed indices by copying rows into the cluster. This proposal serves the same Lance table in place without a copy. The two are complementary. Pull based ingestion suits a hot subset that wants full standard index features, this proposal suits corpora that stay in the lakehouse, and one table can feed both.

Iceberg integration proposal. Iceberg carries no search index of its own and fits the composite engine shape. Lance carries its own and takes the standalone form. Iceberg is analytics oriented, Lance covers analytics and search on the same table with multimodal payloads. Shared abstractions such as a namespace layer or a common cache tier are worth aligning across the two integrations.

k-NN plugin. This proposal is a table format integration, not a vector engine. Lance carries vector, scalar, and FTS indices as first class table objects, and its columnar layout feeds aggregations, so search and analytics run on the same table. The k-NN plugin serves in-memory HNSW on classic Lucene indices, which remains the right choice for latency critical small corpora.

Future work

Dynamic partitioning. The shard count is fixed at attach time. Tracking table growth by rebalancing the partition count at runtime is deferred. Two directions are on the table.

The first is a small OpenSearch core change to let index.number_of_shards become dynamic for an engine that declares itself resize capable, followed by a Lance-side metadata reshard (no data movement). This keeps the standard shard model and every existing OpenSearch client and tool works unchanged.

The second is a shard-free path where a custom distributed executor dispatches fragments directly to data nodes. This sidesteps the cluster state pressure of holding many shards and matches the multi engine direction, at the cost of stepping outside the standard search coordination.

Which of the two is picked, and whether both live side by side, is a separate design.

Native ingestion. The natural second stage accepts document writes through the standard document APIs, making OpenSearch the streaming ingestion layer for the same shared table. Lance's MemWAL specification defines a structure close to an OpenSearch shard with its indexing buffer and translog. That stage adds write owning ingestion shards alongside the stateless search shards of this proposal and deserves its own RFC. Nothing here forecloses it, and the shared table, the index types, and the stateless search shards carry over unchanged.

Full Lucene text features. Committing a Lucene inverted index into the table as a custom index type, with a docid to row address column and coverage tracked in the manifest, would add the Lucene only parts of the text query surface (span and interval queries, offset based highlighting) that the FTS modes do not cover. The Lance index format's unknown type handling permits this, and the design is drafted but deferred.

Related component

Search

Describe alternatives you've considered

File format only integration. Integrating only the Lance file format, behind the dataformat plugin surface as Parquet is, would improve random access for search style row fetches. Rejected because the value lives in the layers above. Lance's vector, scalar, and FTS indices are defined at the table format layer, so a file only integration cannot use any of them. The table services OpenSearch would otherwise reimplement privately (file generation tracking, deletion files, column evolution, index to data consistency) come for free from the Lance libraries. And a pile of .lance files under OpenSearch private metadata is not a Lance table, so external engines cannot read it, forfeiting the interoperability that motivates the integration.

Additional context

• Lance format specification
• Lance MemWAL specification
• Lance index format and unknown type handling
• OpenSearch #20644 Pluggable Component Bundling RFC
• OpenSearch sandbox: dataformat and DataFusion engine work
• sql#5246: SQL/PPL language support for Analytics engine integration
• How LanceDB Accelerates Vector Search at 10 Billion Scale (LanceDB blog)
• lance-ray: distributed index building and compaction for Lance with Ray
• OpenSearch #22594 Arrow format as pull-based ingestion source

No repository files or tests are named. Start by reviewing the RFC's Lance table fundamentals and runtime overview alongside the linked Lance table and index specifications, then clarify the scope of the distributed query layer, index builder, external-write visibility, and optimistic-concurrency behavior. Done requires an agreed implementation plan for this cross-cutting integration.