bug: BM25 FTS query attempts ~96 GiB allocation and aborts in `InvertedIndex::bm25_search_modern_candidates` (regression vs lance 10)

Description

Summary

A BM25 full-text search query (no writes involved) against a jieba-tokenized inverted index triggers a single Rust allocation of 103,079,215,056 bytes (~96 GiB) in lance_index::scalar::inverted::InvertedIndex::bm25_search_modern_candidates, causing an immediate abort().

The same table + index + query works fine on lance 10 (peak RSS ~350 MB). This is a size-calculation bug (integer overflow / wrong memory estimate), not an actual OOM.

Environment

Component	Working	Failing
@lancedb/lancedb (node)	0.37.1	0.38.0
lance core (Rust)	10.0.0	11.0.0
jieba-rs	0.10.1	0.10.1 (same)
pylancedb	0.37.1 (lance 10) → ✅	—
Node.js	22.19.0, linux x64	same
Table size	1,041 rows, FTS index on one text column, vector column 768-dim	same
Tokenizer	jieba, custom dicts under LANCE_LANGUAGE_MODEL_HOME (dict.txt ~5 MB)	same

Table and FTS index were built by the Python SDK; the node SDK only reads. (Cross-language reads are fine — the crash is purely on the FTS query path.)

Minimal repro (node)

import * as lancedb from "@lancedb/lancedb";

// env: LANCE_LANGUAGE_MODEL_HOME points to dir containing jieba/custom/dict.txt + jieba/default/dict.txt
const conn = await lancedb.connect("/path/to/db");
const table = await conn.openTable("documents");

// Pure FTS query alone is enough to crash on node 0.38.0 / lance 11
const res = await table.search("NGP渗透率", "fts").limit(5).toArray();  // ← abort here

Hybrid (search(vec, "vector").fullTextSearch(text).rerank(RRFReranker)) crashes identically — the abort happens in the FTS segment before RRF runs.

Note: the Rust abort() bypasses JS try/catch entirely and dumps core — in a memory-constrained container it presents as an abrupt OOM-kill of the whole process group, with no catchable error surfaced to JS.

Error output (with RUST_BACKTRACE=full)

[0] baseline RSS=80MB
[1] openTable RSS=90MB
memory allocation of 103079215056 bytes failed
   9: alloc::raw_vec::handle_error
  10: <lance_index::scalar::inverted::index::inverted_index::InvertedIndex>
        ::bm25_search_modern_candidates::{closure#0}::{closure#0}::{closure#1}
  11: <InvertedIndex>::bm25_search_modern_candidates::{closure#0}
  12: <InvertedIndex>::bm25_search_documents::{closure#0}::{closure#0}
  13: lance::io::exec::fts::search_segments::{closure#0}
  14: <lance::io::exec::fts::MatchQueryExec as datafusion_physical_plan::ExecutionPlan>::execute
  ...
  26: napi::tokio_runtime::execute_future_impl (node binding)

The requested allocation 103079215056 bytes is near-exactly 96 GiB (0x1800000000) — strongly suggesting an overflowed/incorrect size computation, since the actual data is ~1k rows and a 5 MB dictionary.

Control experiment (same process, same index, same query)

Path	lance 10 (node 0.37.1 / py 0.37.1)	lance 11 (node 0.38.0)
open table	✅ 88–90 MB	✅ 90 MB
pure FTS query	✅ 176 MB, ~0.7 s	💥 abort (~96 GiB alloc)
hybrid + RRF	✅ 246–341 MB	💥 abort (same frame)
hybrid + where filter	✅	—
py SDK same query	✅ 363 MB peak	—

Downgrading only the node SDK to 0.37.1 (lance 10, same jieba-rs 0.10.1) makes every query pass, so the regression is in lance 10 → 11.

Expected behavior

BM25 FTS query should complete in tens of MB, as it does on lance 10.

Suspected area

lance-index InvertedIndex::bm25_search_modern_candidates (new in lance 11's "modern candidates" BM25 path) — the size passed to the allocator appears to be derived from something like doc_count × ... with a wrong/overflowed factor rather than actual posting-list sizes.

Steps to reproduce

import * as lancedb from "@lancedb/lancedb";

// env: LANCE_LANGUAGE_MODEL_HOME points to dir containing jieba/custom/dict.txt + jieba/default/dict.txt
const conn = await lancedb.connect("/path/to/db");
const table = await conn.openTable("documents");

// Pure FTS query alone is enough to crash on node 0.38.0 / lance 11
const res = await table.search("NGP渗透率", "fts").limit(5).toArray();  // ← abort here

Expected behavior

No response

Lance version

11.0.0

Language binding

Java

Environment

No response

Logs / traceback

Start with lance-index's InvertedIndex::bm25_search_modern_candidates entry point and run the minimal Node reproduction against lance 11, comparing it with lance 10. Trace the allocation-size calculation for the jieba-tokenized index and confirm that the BM25 query completes without the ~96 GiB allocation or abort.