[C++][Python] Support parsing a StringArray full of JSON to a Table
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Working with pyarrow, there doesn't seem to be any way to parse a `StringArray` full of json into an `Array` or `Table` of nested data. This surprises me, becuase [pyarrow.json.read_json](https://arrow.apache.org/docs/python/json.html) does exactly the right thing... but only for line-delimited json files. At least, I didn't see anything e.g. in [pyarrow.compute](https://arrow.apache.org/docs/python/compute.html) and a google search came up empty.
Looking deeper there doesn't seem to be anything on the C++ side, either.
Skimming the C++ sources, I think there's a core logic that could be wrapped up into a proper compute function. If I read correctly:
* [TableReaderImpl::Read](https://github.com/apache/arrow/blob/master/cpp/src/arrow/json/reader.cc#L268)
* calls [TableReaderImpl::ParseAndInsert](https://github.com/apache/arrow/blob/master/cpp/src/arrow/json/reader.cc#L287)
* calls [ParseBlock](https://github.com/apache/arrow/blob/master/cpp/src/arrow/json/reader.cc#L159)
* which uses a [HandlerBase](https://github.com/apache/arrow/blob/master/cpp/src/arrow/json/parser.cc#L649) instance that implements `BlockParser`
* whose [doParse method](https://github.com/apache/arrow/blob/master/cpp/src/arrow/json/parser.cc#L773) uses a rapidjson `Reader` to do the heavy lifting.
I _think_ to parse a `StringArray` (instead of a file), we'd just need an `ArrayParser` variant, similar to `BlockParser`, that presents rapidjson a different buffer for each string to be parsed --- maybe a [std::spanstream](https://en.cppreference.com/w/cpp/io/basic_ispanstream) --- so that EOF becomes the "delimiter" instead of newline?
### Component(s)
C++, Python
Contributor guide
Research direction
Start by reading cpp/src/arrow/json/reader.cc and parser.cc, following TableReaderImpl::Read, ParseAndInsert, ParseBlock, HandlerBase, and BlockParser::doParse. Define and implement the requested C++ and Python-facing behavior so a StringArray containing JSON can produce nested Array or Table results, then verify the behavior with focused tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100