bug: FTS on a JSON subfield — path-scoped inverted index is silently inert, and non-triple queries panic

Summary

FTS over a string nested inside a JSON column is possible and gives accurate, field-scoped answers — but only through a direct INVERTED index queried with an undocumented field,type,value triple syntax. The two things a user would naturally try both fail:

1. A JSON-path inverted index is accepted, listed, and completely inert. IndexConfig(index_type="json", parameters={"target_index_type": "inverted", "path": "$.title"}) builds without error and shows up in list_indices(), but the planner never uses it — the plan is FlatMatchQuery, byte-for-byte identical to having no index at all. Queries fall back to a brute-force scan of the raw JSON text, so they match JSON key names and values from other fields, i.e. they ignore the path the index was declared on.
2. Any non-triple query against a whole-column INVERTED index panics. flatten_triplet(query_text, ...).unwrap() at rust/lance-index/src/scalar/inverted/tokenizer/document_tokenizer.rs:147 unwraps a Result whose error is driven purely by user input, so the ordinary FTS query "brown" panics a background thread and surfaces as RuntimeError: Task was aborted.

Repro

pylance 10.0.0, Linux x86_64.

import os, shutil, tempfile, json, logging
import pyarrow as pa, lance
from lance.indices import IndexConfig
from lance.query import MatchQuery
logging.disable(logging.WARNING)

DOCS = [{"title": "quick brown fox", "author": "alice"},
        {"title": "lazy dog sleeps", "author": "bob"},
        {"title": "brown bear roars", "author": "carol"},
        {"title": "fox and hound",    "author": "alice"}]
docs = [json.dumps(d) for d in DOCS]
jf = pa.field("doc", pa.string(), metadata={b"ARROW:extension:name": b"arrow.json"})
schema = pa.schema([pa.field("id", pa.int32()), jf])

def mk(cfg=None, name="fts"):
    uri = os.path.join(tempfile.mkdtemp(), "f.lance"); shutil.rmtree(uri, ignore_errors=True)
    ds = lance.write_dataset(pa.table({"id": pa.array(range(4), pa.int32()),
        "doc": pa.array(docs, pa.string())}, schema=schema), uri)
    if cfg is not None:
        ds.create_scalar_index("doc", cfg, name=name)
    return lance.dataset(uri)

def search(ds, q):
    try:
        return sorted(ds.scanner(columns=["id"], full_text_query=MatchQuery(q, column="doc")
                      ).to_table().column("id").to_pylist())
    except Exception as e:
        return f"ERR {type(e).__name__}: {str(e)[:55]}"

def plan(ds, q="brown"):
    return next((l.strip() for l in ds.scanner(columns=["id"],
        full_text_query=MatchQuery(q, column="doc")).explain_plan(True).splitlines()
        if "MatchQuery" in l), "?")

NONE = mk()
PATH = mk(IndexConfig(index_type="json",
      parameters={"target_index_type": "inverted", "path": "$.title"}), "fts_title")
print("Defect 1: JSON-path inverted index (path=$.title)")
print("  created:", [(i["name"], i["type"]) for i in PATH.list_indices()])
for term, want in [("brown", [0,2]), ("fox", [0,3]), ("alice", []), ("title", []), ("author", [])]:
    print(f"  {term:<8} {str(search(NONE, term)):<14} {str(search(PATH, term)):<16} {want}")
print("  plan, no index  :", plan(NONE))
print("  plan, path index:", plan(PATH))

WHOLE = mk("INVERTED", "fts_all")
print("\nDefect 2: direct INVERTED on the JSON column")
print("  plan:", plan(WHOLE))
for q in ["brown", "title,string,brown", "title,str,brown"]:
    print(f"  {q!r:<24} -> {search(WHOLE, q)}")
for q, want in [("title,str,brown", [0,2]), ("title,str,alice", []), ("author,str,alice", [0,3])]:
    got = search(WHOLE, q)
    print(f"  {q!r:<24} -> {got}   want {want}   {got == want}")

Actual

Defect 1: JSON-path inverted index (path=$.title)
  created: [('fts_title', 'Json')]
  term     no index       with path index  want (for $.title)
  brown    [0, 2]         [0, 2]           [0, 2]
  fox      [0, 3]         [0, 3]           [0, 3]
  alice    [0, 3]         [0, 3]           []
  title    [0, 1, 2, 3]   [0, 1, 2, 3]     []
  author   [0, 1, 2, 3]   [0, 1, 2, 3]     []
  plan, no index  : FlatMatchQuery: column=doc, query=brown
  plan, path index: FlatMatchQuery: column=doc, query=brown

Defect 2: direct INVERTED on the JSON column
  plan: MatchQuery: column=doc, query=[brown]
  'brown'                  -> ERR ArrowInvalid: External error: RuntimeError: Task was aborted
  'title,string,brown'     -> ERR ArrowInvalid: External error: RuntimeError: Task was aborted
  'title,str,brown'        -> [0, 2]
  'title,str,alice'        -> []          want []          True
  'author,str,alice'       -> [0, 3]      want [0, 3]      True

Defect 1 detail

alice appears only in $.author, yet matches rows 0 and 3. title and author are JSON key names, not content, yet each matches all four rows. Every column is identical to the no-index run, and both plans are FlatMatchQuery, so the index contributes nothing — the answers come from a flat scan over the serialized JSON text.

So a user who builds this index gets no acceleration and no path scoping, with no indication that either is missing. Note also that list_indices() reports the index type as Json rather than Inverted, which matches the existing TODO on JsonIndex::index_type().

Defect 2 detail

Both panics come from the same .unwrap():

fn token_stream_for_search<'a>(&'a mut self, query_text: &'a str) -> BoxTokenStream<'a> {
    let tokens = flatten_triplet(query_text, &mut self.tokenizer).unwrap();   // :147

• "brown" → InvalidInput { source: "Invalid triple format: brown" } (:180)
• "title,string,brown" → InvalidInput { source: "Invalid triple type: string" } (:214)

The second is worth calling out: the type token is str, not string, and getting it wrong panics rather than erroring. Accepted tokens are str, number, bool, null; field names are dotted paths built by flatten_json (so meta.tag,str,nature works for a nested object).

Expected

• A JSON-path inverted index should either be used by the planner and scoped to its path, or be rejected at creation time with a clear "not supported" error. Silently creating an index that is never consulted, and answering with whole-document semantics under a $.title declaration, is the worst outcome.
• A malformed FTS query should return an InvalidInput error to the caller, not panic a background thread. The error type is already there; it just needs to propagate instead of being unwrapped.

Notes

• Verified against a control: a plain string column with INVERTED plans as MatchQuery: column=title and returns correct results, so the FlatMatchQuery fallback is specific to the JSON path index.
• The triple syntax appears to be the intended query interface for the JSON inverted index, but I could not find it documented anywhere. Given #7445 is already reworking flattened JSON sub-doc indexing, it may be worth settling the user-facing query surface there — ideally so that MatchQuery("brown", column="doc.title") or similar works instead of requiring callers to hand-assemble title,str,brown.
• Related: #4749 (add full text json index), #7445 (flattened JSON sub-doc indexing), #8806 (JSON path scalar index returns incorrect results for json_extract filters).

Run the provided Python reproduction, then inspect rust/lance-index/src/scalar/inverted/tokenizer/document_tokenizer.rs around line 147 and the JSON-index planning path. Verify how malformed triple queries and JSON-path indexes are handled. Done means malformed input returns InvalidInput without aborting a background task, and JSON-path indexes are either used with path scoping or rejected clearly.