apache / apache/datafusion

Support top-level arrays when reading JSON files

Open
#17,223 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Datafusion can't read JSONL/ND-JSON files where the rows are arrays rather than objects. e.g.:

```
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
```

### Describe the solution you'd like

It would be great if Datafusion could directly consume ND-JSON files containing top-level arrays, mapping the array to a single field with a customisable name, like data or items.

I notice that the underlying arrow library seems to support this concept via `datafusion::arrow::json::ReaderBuilder::new_with_field`, but this doesn't seem to be exposed by the JSON `FileFormat` that Datafusion exposes.

### Describe alternatives you've considered

At present I am working around the issue with a pre-processing step that wraps each line with a { "data": ... } wrapper, however this comes with a heavy performance penalty.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing DataFusion's JSON FileFormat and how it constructs the Arrow ReaderBuilder; compare that path with datafusion::arrow::json::ReaderBuilder::new_with_field. Done means ND-JSON lines containing top-level arrays can be read directly, with the array mapped to a configurable field name such as data or items.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.