bug: JSON index on a nested path is matched by typed accessors that cannot evaluate it, so indexed and unindexed results disagree

Summary

A JSON scalar index trained on a multi-segment JSONPath is matched by the typed accessors (json_get_int / json_get_float / json_get_bool / json_get_string) when the filter spells the path identically — but those accessors do not evaluate JSONPath. The result is that the indexed and unindexed evaluations of the same predicate disagree, silently.

This is the same family as #8806 but a distinct trigger. #8806 is about json_extract evaluating to serialized JSON text, and explicitly notes that json_get_string(meta, 'kind') does not match a $.kind index and is therefore correctly left as an unindexed scan. The case here is the one where the spellings do match.

Mechanism

The build side and the query side use different path languages.

• Build accepts full JSONPath: json_extract_with_type -> extract_json_path_with_type -> common::parse_json_path -> jsonb::jsonpath::parse_json_path (rust/lance-datafusion/src/udf/json.rs). Upstream tests index "$.age" and "$.items[*].price".
• Query does not: json_get_int and friends call common::get_json_value_by_key (rust/lance-datafusion/src/udf/json.rs), a single-level get_by_name(key) (or an array index when the root is an array). It never parses JSONPath.
• Matching is plain string equality: JsonQueryParser::is_valid_reference (rust/lance-index/src/scalar/json.rs) compares the filter's key literal against the stored path. Nothing rejects a multi-segment spelling.

So for an index built with {"target_index_type": "btree", "path": "$.user.age"} and the filter json_get_int(doc, '$.user.age') = 30:

• is_valid_reference matches on string equality, and the index answers the predicate with real rows.
• The unindexed evaluation of the same predicate looks for a literal top-level key named $.user.age, finds none, and yields NULL for every row.

The two paths return different answers for the same filter. As in #8806 the plan has no recheck, so the disagreement is silent.

The only path spellings where the two languages agree are bare top-level keys ("age"), and — by how jsonb's parser handles them — plain array indices.

Note on coverage

The regression test added for #8806 (rust/lance/src/dataset/tests/dataset_index.rs, asserting indexed results equal a use_scalar_index(false) baseline) guards json_extract only. Nested paths on the typed accessors are not covered by it.

Found by code reading while building JSON-path index support in a downstream system; I have not run a Python repro, so please treat the exact spelling above as illustrative rather than a verified script.

Suggested fix

Either make is_valid_reference reject a stored path that the typed accessors cannot evaluate (anything beyond a bare key or array index), or give the typed accessors real JSONPath evaluation so the two sides agree. The first is the conservative option and would turn a silent wrong answer into a correct unindexed scan.

Extending the #8806 baseline-comparison test to cover nested paths on the typed accessors would catch this class.

Start with rust/lance-index/src/scalar/json.rs and inspect JsonQueryParser::is_valid_reference alongside the typed accessor paths in rust/lance-datafusion/src/udf/json.rs. Read rust/lance/src/dataset/tests/dataset_index.rs and extend its indexed-versus-unindexed comparison to cover nested paths on typed accessors. Done means the indexed and unindexed evaluations agree without silently returning different results.