lance-format / lance-format/lance

feat(index): support flattened JSON sub-doc indexing

Open
#7,445 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Parent: #4516
Draft PR: #7377

Motivation

Lance JSON inverted indexes should preserve array/object structure so path queries over arrays of objects can distinguish exact positions, wildcards, and nested-object constraints.

Tantivy documents a known flaw of bag-of-terms JSON array indexing: a query like cart.product_type:sneakers AND cart.attributes.color:red can match even when sneakers and red live in different objects in the same array. Lance's current single-token-stream JSON index has the same class of problem.

Proposal

Introduce FlattenedSubDocs mode for new JSON inverted indexes. A JSON row can flatten into multiple internal sub-documents, and matches are mapped back to the original Lance row id. This lets related JSON path constraints match inside the same flattened array element before row-id remapping.

Also add disable_cross_array_unnest=false by default, with an opt-in memory-safety tradeoff for documents with many sibling arrays, following Pinot's DisableCrossArrayUnnest behavior.

Expected behavior

  • Existing JSON indexes remain readable using SingleDocument mode.
  • New JSON indexes use FlattenedSubDocs mode.
  • Positional and wildcard queries can return different row sets, for example foo[0].bar[0],str,y vs foo[0].bar[*],str,y.
  • Nested-object constraints should be expressed as one JSON MatchQuery with semicolon-separated triplets plus Operator::And, so matching happens at flattened sub-doc granularity.
  • Separate BooleanQuery.must child queries remain row-level composition and are not the query shape for nested-object semantics.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the parent issue #4516 and the draft PR #7377, then inspect the existing JSON inverted-index behavior described here. Done means new indexes use FlattenedSubDocs, existing indexes remain readable, positional and wildcard queries return the expected row sets, and nested-object constraints match within the same flattened sub-document.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.