bug: JSON integers above i64::MAX break JSON index creation and typed reads; numbers above u64::MAX are silently truncated to f64

Correction (see comment below): the "stored losslessly" framing here holds only up to u64::MAX. Numbers above u64::MAX, and exact decimals, are silently truncated to f64 at write time, because Lance depends on jsonb with default-features = false and so compiles out its Decimal64/Decimal128/Decimal256 support. That is a second, worse defect than the one described below.

Summary

A JSON document containing an integer between i64::MAX and u64::MAX is written and stored losslessly, but every typed path over it breaks: create_scalar_index for a JSON BTREE index is rejected outright, json_get_int fails the whole query, and when the index does happen to build it silently degrades the value to f64.

Root cause is that the JSON type layer has no unsigned 64-bit representation. JsonbType (rust/lance-datafusion/src/udf/json.rs) only has Int64 and Float64, so extract_json_path_with_type tags a jsonb Number::UInt64 as Float64:

} else if raw.is_number().unwrap_or(false) {
    let is_float_storage = matches!(raw.as_number(), Ok(Some(jsonb::Number::Float64(_))));
    if !is_float_storage && raw.is_i64().unwrap_or(false) {
        JsonbType::Int64
    } else {
        JsonbType::Float64   // <-- u64 above i64::MAX lands here
    }
}

JsonIndexPlugin::extract_json_with_type_info then infers the index's Arrow type from only the first non-null value at path, and convert_stream_by_type hard-errors on any later value that does not decode to that type. So whether indexing succeeds, fails, or silently loses precision depends on the order of the rows.

This is the same i64-or-f64 class of bug as #7910 (SQL filters rejecting UInt64 literals above i64::MAX), which was fixed for the expression planner but not for the JSON functions or the JSON index.

Repro

pylance 10.0.0, Linux x86_64.

import os, shutil, tempfile
import pyarrow as pa
import lance
from lance.indices import IndexConfig

BIG = 2**63          # i64::MAX + 1
meta = pa.field("meta", pa.string(), metadata={b"ARROW:extension:name": b"arrow.json"})
schema = pa.schema([pa.field("id", pa.int32()), meta])

def build(docs):
    uri = os.path.join(tempfile.mkdtemp(), "j.lance")
    shutil.rmtree(uri, ignore_errors=True)
    return lance.write_dataset(pa.table(
        {"id": pa.array(range(len(docs)), pa.int32()),
         "meta": pa.array(docs, pa.string())}, schema=schema), uri)

def index(ds):
    try:
        ds.create_scalar_index("meta", IndexConfig(index_type="json",
            parameters={"target_index_type": "btree", "path": "$.v"}), name="idx_v")
        return "OK"
    except Exception as e:
        return f"{type(e).__name__}: {str(e)[:110]}"

print("1) storage is lossless")
ds = build([f'{{"v": {BIG}}}', f'{{"v": {2**64 - 1}}}'])
print("   stored      :", ds.to_table().column("meta").to_pylist())
print("   json_extract:", ds.scanner(columns={"v": "json_extract(meta, '$.v')"}).to_table().column("v").to_pylist())

print("\n2) index creation is rejected, and it depends on row order")
print("   in-range int first :", index(build(['{"v": 1}', f'{{"v": {BIG}}}'])))
print("   out-of-range first :", index(build([f'{{"v": {BIG}}}', '{"v": 1}'])))

print("\n3) when it does build, large ints are silently conflated as f64")
ds = build([f'{{"v": {BIG}}}', f'{{"v": {BIG + 1}}}'])
print("   index          :", index(ds))
print("   json_get_float :", ds.scanner(columns={"v": "json_get_float(meta, 'v')"}).to_table().column("v").to_pylist())

print("\n4) json_get_int fails the whole query")
ds = build([f'{{"v": {BIG}}}'])
try:
    print("  ", ds.scanner(columns={"v": "json_get_int(meta, 'v')"}).to_table().column("v").to_pylist())
except Exception as e:
    print(f"   {type(e).__name__}: {str(e)[:110]}")

Actual

1) storage is lossless
   stored      : ['{"v":9223372036854775808}', '{"v":18446744073709551615}']
   json_extract: ['9223372036854775808', '18446744073709551615']

2) index creation is rejected, and it depends on row order
   in-range int first : ValueError: Invalid user input: Failed to deserialize JSONB to i64 at index 1: UnexpectedType
   out-of-range first : OK

3) when it does build, large ints are silently conflated as f64
   index          : OK
   json_get_float : [9.223372036854776e+18, 9.223372036854776e+18]

4) json_get_int fails the whole query
   ArrowInvalid: External error: Query Execution error: Execution error: Failed to convert to integer: InvalidCast

Four separate consequences:

1. Storage is fine in this range. The write succeeds and the raw JSON round-trips exactly, up to u64::MAX (but not beyond it — see the correction above). json_extract returns the exact digits, since it hands back serialized JSON text. So the data on disk is correct — only the typed layer above it is broken.
2. create_scalar_index is rejected with Failed to deserialize JSONB to i64 at index 1: UnexpectedType (rust/lance-index/src/scalar/json.rs:585). Whether this fires is order-dependent: the type is inferred from the first non-null value, so a dataset whose first row is a small int fails, while the same values in the opposite order build fine. Adding one out-of-range row to an otherwise in-range column makes the index unbuildable.
3. Silent precision loss when the out-of-range value happens to come first: the index is built as Float64, so 2**63 and 2**63 + 1 both become 9.223372036854776e+18 and are indistinguishable to an equality or range query.
4. json_get_int fails the entire query rather than returning null or an error for just the offending row (Failed to convert to integer: InvalidCast).

Expected

An integer in the u64 range should either be supported end-to-end by the JSON functions and the JSON index, or be rejected at write time with a clear message — not accepted, stored losslessly, and then made unqueryable through the typed accessors.

At minimum, the index's inferred type should not depend on which row happens to be first, and a value that cannot be represented in the inferred type should not abort index construction for the whole dataset.

Notes

• A UInt64 variant in JsonbType (with the corresponding DataType::UInt64 mapping and decode branch in convert_stream_by_type) would cover the positive u64 range. Note that negatives below i64::MIN are already silently floated at write time ({"v": -9223372036854775809} stores as -9.223372036854776e+18), so the two ends are not symmetric. Enabling jsonb's arbitrary_precision feature would raise the exact range much further in both directions — see the correction above.
• Type inference over only the first non-null value is fragile beyond this bug — any heterogeneously-typed JSON path hits the same hard error. Widening to a common type across the scan, or falling back to the (lenient) Utf8 branch, would make index construction order-independent.
• JsonIndex::statistics() is todo!(), so dataset.stats.index_stats(...) raises PanicException: not yet implemented on any JSON index, which makes this class of problem hard to diagnose from the Python side.
• Filed separately/related: JSON-path index queries return incorrect results for json_extract filters (#8806). The failures in step 3 above are compounded by that issue.