lance-format / lance-format/lance
feat: Add typed JSONPath extraction functions
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Problem
Lance currently has two different JSON access patterns:
json_get_string,json_get_int,json_get_float, andjson_get_boolaccept one literal object key or array index and return a typed scalar.json_extractaccepts JSONPath, but returns the extracted value as serialized JSON text.
As a result, reading a typed nested scalar requires composing json_get calls:
json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name')
Passing a dotted path such as json_get_string(data, 'user.profile.name') cannot safely be reinterpreted as JSONPath because JSON object keys may legitimately contain dots.
Proposed contract
Add typed JSONPath extraction functions, for example:
json_extract_string(data, '$.user.profile.name')
json_extract_int(data, '$.user.age')
json_extract_float(data, '$.metrics.score')
json_extract_bool(data, '$.flags.active')
The exact names are open to design, but the API should:
- use the same JSONPath syntax and evaluator as
json_extract; - return typed Arrow scalar values;
- preserve the current literal-key semantics of
json_get_*; - define behavior for a missing path, JSON
null, incompatible types, nested objects, and arrays; - define whether conversions match the existing
json_get_*coercion rules.
Acceptance criteria
- Typed extraction works for nested object fields and array elements.
- Missing paths and JSON
nullhave documented SQL null behavior. - Type mismatches and coercions are consistent and covered by tests.
- A literal key containing
.remains accessible throughjson_get_*without ambiguity. - DataFusion UDF tests and Python SQL/filter integration tests cover the new functions.
- User-facing JSON query documentation explains when to use
json_get_*,json_extract, and the typed JSONPath functions.
Related work
- JSON support epic: https://github.com/lance-format/lance/issues/4516
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing json_get_* and json_extract implementations, the DataFusion UDF tests, and the Python SQL/filter integration tests. Compare their JSONPath evaluation, scalar typing, null handling, and coercion behavior before choosing the function names and contract. Done means nested objects and arrays work, literal dotted keys remain supported, documented SQL null and mismatch behavior is covered by tests, and the user-facing JSON query documentation is updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- api, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100