datafusion-contrib / datafusion-contrib/datafusion-functions-json

Getting arrays of objects

Open
#13 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
62
Forks
33
Avg merge
1d 12h
Merged PRs (30d)
9

Description

Thanks for putting this together, this is a great improvement on the state of JSON handling in datafusion! I've been trying this out in Arroyo (https://github.com/ArroyoSystems/arroyo/pull/640), and it's definitely a big perf improvement on what we've been doing.

One limitation I've encountered is dealing with arrays of objects. For example, something like

{
  "a": [
    {"b": 5},   
    {"b": 9},
    {"b": 3}
  ]
}

A common pattern in Arroyo is to extract an array from JSON data, then call our unnest operator on it (which unrolls a list into individual rows). For that, we need a way to get an Arrow list out of the data. But there's no function that's capable of returning an array.

Two options I've considered:

  • Some syntax to "map" over a list in the json path (like json_get_int(s, 'a', *, 'b')), where the * represents an array that we're mapping over; this would return a SQL array/arrow list [5, 9, 3]
  • A get_json_array function, which returns an array of JsonUnion objects, which could then be queried without needing to be deserialized again (like get_json_int(unnest(get_json_array(s, 'a')), 'b)).

I've got most of a working prototype of the second one, but I'd be interested in thoughts on what approach makes sense (or whether it's within the interest of this project to support more complex JSON structures like this).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the existing JSON extraction functions and the Arroyo PR 640 context first. Compare the proposed JSON-path mapping approach with a get_json_array function returning JsonUnion objects, including how either would work with Arroyo's unnest operator. Done means the project has agreed on an approach and its intended array-of-objects behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.