[Python][C++] Flexible handling of conflicts during JSON type inference
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
JSON is extremely flexible, and arrow's [automatic type inference rules](https://arrow.apache.org/docs/python/json.html#automatic-type-inference) often fails to find a schema.
Here is a simple example on which `pyarrow.json.read_json` returns `ArrowInvalid: JSON parse error: Column(/lol/[]) changed from number to array in row 0`
```json
{ "lol": [42, []]}
```
I'm attempting to help solve [this issue](https://github.com/huggingface/datasets/issues/5950) which is basically to cope with such "unstructured" JSON formats. AFAICT Arrow doesn't make this easy, because parsing/conversion simply fails without an obvious way to move forward. It would be nice if there was a way to proceed even if type inference fails. Here are a few ideas:
* If type inference fails, optionally use a general container type (`JsonType`?) instead. This could perhaps be a ParseOption passed to `read_json`.
* Allow the user to augment (e.g., a callback) the type inference process, especially when it would otherwise fail. Again, this could be a ParseOption.
* Expose the type inference algorithm as a separate function, with a way to explicitly indicate failures in the schema. In other words, don't throw an exception, but rather mark that part of the schema in some way as needing additional input from the caller.
If I missed an obvious way of coping with these unstructured JSON files, please let me know!
### Component(s)
Python, Other
Contributor guide
Assessment
This issue has not been assessed yet.