datafusion-contrib / datafusion-contrib/datafusion-functions-json
Getting arrays of objects
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 62
- Forks
- 33
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 9
Description
Thanks for putting this together, this is a great improvement on the state of JSON handling in datafusion! I've been trying this out in Arroyo (https://github.com/ArroyoSystems/arroyo/pull/640), and it's definitely a big perf improvement on what we've been doing.
One limitation I've encountered is dealing with arrays of objects. For example, something like
{
"a": [
{"b": 5},
{"b": 9},
{"b": 3}
]
}
A common pattern in Arroyo is to extract an array from JSON data, then call our unnest operator on it (which unrolls a list into individual rows). For that, we need a way to get an Arrow list out of the data. But there's no function that's capable of returning an array.
Two options I've considered:
- Some syntax to "map" over a list in the json path (like
json_get_int(s, 'a', *, 'b')), where the*represents an array that we're mapping over; this would return a SQL array/arrow list[5, 9, 3] - A
get_json_arrayfunction, which returns an array of JsonUnion objects, which could then be queried without needing to be deserialized again (likeget_json_int(unnest(get_json_array(s, 'a')), 'b)).
I've got most of a working prototype of the second one, but I'd be interested in thoughts on what approach makes sense (or whether it's within the interest of this project to support more complex JSON structures like this).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the existing JSON extraction functions and the Arroyo PR 640 context first. Compare the proposed JSON-path mapping approach with a get_json_array function returning JsonUnion objects, including how either would work with Arroyo's unnest operator. Done means the project has agreed on an approach and its intended array-of-objects behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100