[Python] JSON Type Inference + datasets: `null` doesn't fallback to other type between dataset files
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
When inferring types in a JSON dataset, `null` doesn't fallback to other types when those types are across files in the dataset.
For example, in a JSON lines dataset like so:
File 1
> {"userId":null}
File 2
> {"userId": null}
> {"userId": "example-string"}
Results in an error:
> Invalid: JSON parse error: Column(/userId) changed from null to string in row 1
However, [the documentation states](https://arrow.apache.org/docs/python/json.html#automatic-type-inference) that "JSON `null` values convert to the null type, but can fall back to any other type."
So this fallback works _within the file_:
File 1
> {"userId":null}
> {"userId": "example"}
File 2
> {"userId":"for-example"}
This dataset works! Resulting in:
> pyarrow.Table
> userId: string
Version & Platform: pyarrow 14.0.1 and 15.0.0, Python 3.9.2
Fully runnable reproduction:
```py
import tempfile
import pathlib
import pyarrow.dataset as ds
base = pathlib.Path(tempfile.mkdtemp(prefix="pyarrow-"))
(base / "json_dataset").mkdir(exist_ok=True)
# Make file 1
f = open(base / "json_dataset/data1.json", "a")
f.write('{"userId": null}\n{"userId": null}')
f.close()
# Make file 2
f = open(base / "json_dataset/data2.json", "a")
f.write('{"userId": null}\n{"userId": "example-string"}')
f.close()
# writing it into two parquet files
table = ds.dataset(base / "json_dataset", format="json")
print(table.to_table())
```
In this repro, if you change the order of the files, it'll work.
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.