JsonIndexDetails cannot represent target_data_type, so details are insufficient to rebuild a JSON index

Summary

JsonIndexDetails cannot represent the target data type, so the persisted details are not sufficient to rebuild an equivalent JSON index.

message JsonIndexDetails {
  string path = 1;
  google.protobuf.Any target_details = 2;
}

(protos/index.proto)

target_details for the common targets is an empty message — BTreeIndexDetails {} and BitmapIndexDetails {} (protos/index_old.proto). So the entire persisted payload for a JSON index is a path plus the target's type identity.

Meanwhile JsonIndexParameters (rust/lance-index/src/scalar/json.rs) carries four fields:

struct JsonIndexParameters {
    target_index_type: String,
    target_index_parameters: Option<String>,
    target_data_type: Option<JsonIndexTargetType>,
    path: String,
}

target_data_type has no home in the details. Details are a strict subset of params here.

Why it matters

When target_data_type is omitted, JsonIndexPlugin infers the type from the data — infer_type_from_batch returns on the first non-null value's type tag and falls back to Utf8 when everything is null. That makes a rebuild driven purely by details non-deterministic with respect to the data it sees:

• A rebuild over a different row set can infer a different type than the original index.
• A build that is split across workers, each training over a disjoint subset of fragments, can have each worker infer independently. A sparse field is enough to diverge: one subset sees {"v": 1} and infers Int64, another is entirely null at that path and falls back to Utf8.

There is no cross-worker or cross-build coordination on the create path, so nothing reconciles the disagreement.

Note the type is recoverable from a loaded index — JsonIndex::derive_index_params() reconstructs the full parameters including target_data_type via target_index.training_data_type(). The gap is specifically in the persisted details, which is the representation a downstream system would naturally carry and pass through.

Request

Add target_data_type to JsonIndexDetails so the details are a faithful, lossless description of the index, and a rebuild from details reproduces the original schema rather than re-inferring it.

This is the same class as #4628 (JSON index should store the target index version in the details) — the details message is currently missing information required to reconstruct the index.

Related: #5311 (make JsonIndexParameters constructible), which covers the params side of the same problem.