[C++] Have JSON reader treat mixed singleton/array fields as arrays.
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
I frequently encounter real-world files that mix array types and singletons across entries for a single field. For example, consider an ndjson file consisting of:
```java
{"author": "Hunter Thompson"}
{"author": ["Bob Woodward", "Carl Bernstein"]}
```
Widely used specs promote writing JSON like this, where a singleton isn't be wrapped in array brackets. For example, the 'target' field in the w3 annotation model [may be a string or an array of strings.](https://www.w3.org/TR/annotation-model/#:~:text=The%20body%20and/or%20target%20relationships%20of%20the%20Annotation%20may%20be%20arrays%20rather%20than%20a%20single%20object.)
Currently I see no way to read this sort of data with the C++ json reader. It would be nice if arrow's ndjson reader could do two things to support data like this.
1. When inferring types, silently promote entries of type to type if the column is mixed;
1. When passed an explicit schema that includes a ListArray, promote all instances of the field to an array if they aren't already.
My sense is that this might be pretty simple.
Thanks to everyone who works on this project.
**Reporter**: [Ben Schmidt](https://issues.apache.org/jira/browse/ARROW-15978) / @bmschmidt
**Note**: *This issue was originally created as [ARROW-15978](https://issues.apache.org/jira/browse/ARROW-15978). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with Apache Arrow's C++ NDJSON/JSON reader and trace the paths for type inference and explicit ListArray schemas. Verify how mixed singleton and array values are currently handled, then confirm that inferred mixed fields and explicitly list-typed fields both produce arrays consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, json
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100