apache / apache/arrow

[C++][Python] Support parsing a StringArray full of JSON to a Table

Open
#33,662 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Working with pyarrow, there doesn't seem to be any way to parse a `StringArray` full of json into an `Array` or `Table` of nested data. This surprises me, becuase [pyarrow.json.read_json](https://arrow.apache.org/docs/python/json.html) does exactly the right thing... but only for line-delimited json files. At least, I didn't see anything e.g. in [pyarrow.compute](https://arrow.apache.org/docs/python/compute.html) and a google search came up empty.

Looking deeper there doesn't seem to be anything on the C++ side, either.

Skimming the C++ sources, I think there's a core logic that could be wrapped up into a proper compute function. If I read correctly:
* [TableReaderImpl::Read](https://github.com/apache/arrow/blob/master/cpp/src/arrow/json/reader.cc#L268)
* calls [TableReaderImpl::ParseAndInsert](https://github.com/apache/arrow/blob/master/cpp/src/arrow/json/reader.cc#L287)
* calls [ParseBlock](https://github.com/apache/arrow/blob/master/cpp/src/arrow/json/reader.cc#L159)
* which uses a [HandlerBase](https://github.com/apache/arrow/blob/master/cpp/src/arrow/json/parser.cc#L649) instance that implements `BlockParser`
* whose [doParse method](https://github.com/apache/arrow/blob/master/cpp/src/arrow/json/parser.cc#L773) uses a rapidjson `Reader` to do the heavy lifting.

I _think_ to parse a `StringArray` (instead of a file), we'd just need an `ArrayParser` variant, similar to `BlockParser`, that presents rapidjson a different buffer for each string to be parsed --- maybe a [std::spanstream](https://en.cppreference.com/w/cpp/io/basic_ispanstream) --- so that EOF becomes the "delimiter" instead of newline?

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Research direction

Start by reading cpp/src/arrow/json/reader.cc and parser.cc, following TableReaderImpl::Read, ParseAndInsert, ParseBlock, HandlerBase, and BlockParser::doParse. Define and implement the requested C++ and Python-facing behavior so a StringArray containing JSON can produce nested Array or Table results, then verify the behavior with focused tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.