Support top-level arrays when reading JSON files
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Datafusion can't read JSONL/ND-JSON files where the rows are arrays rather than objects. e.g.:
```
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
```
### Describe the solution you'd like
It would be great if Datafusion could directly consume ND-JSON files containing top-level arrays, mapping the array to a single field with a customisable name, like data or items.
I notice that the underlying arrow library seems to support this concept via `datafusion::arrow::json::ReaderBuilder::new_with_field`, but this doesn't seem to be exposed by the JSON `FileFormat` that Datafusion exposes.
### Describe alternatives you've considered
At present I am working around the issue with a pre-processing step that wraps each line with a { "data": ... } wrapper, however this comes with a heavy performance penalty.
### Additional context
_No response_
Contributor guide
Research direction
Start by tracing DataFusion's JSON FileFormat and how it constructs the Arrow ReaderBuilder; compare that path with datafusion::arrow::json::ReaderBuilder::new_with_field. Done means ND-JSON lines containing top-level arrays can be read directly, with the array mapped to a configurable field name such as data or items.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100