apache / apache/arrow

[Python] JSON Type Inference + datasets: `null` doesn't fallback to other type between dataset files

Open
#40,107 2 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When inferring types in a JSON dataset, `null` doesn't fallback to other types when those types are across files in the dataset.

For example, in a JSON lines dataset like so:

File 1
> {"userId":null}

File 2
> {"userId": null}
> {"userId": "example-string"}

Results in an error:
> Invalid: JSON parse error: Column(/userId) changed from null to string in row 1

However, [the documentation states](https://arrow.apache.org/docs/python/json.html#automatic-type-inference) that "JSON `null` values convert to the null type, but can fall back to any other type."

So this fallback works _within the file_:

File 1
> {"userId":null}
> {"userId": "example"}

File 2
> {"userId":"for-example"}

This dataset works! Resulting in:

> pyarrow.Table
> userId: string

Version & Platform: pyarrow 14.0.1 and 15.0.0, Python 3.9.2

Fully runnable reproduction:

```py
import tempfile
import pathlib
import pyarrow.dataset as ds

base = pathlib.Path(tempfile.mkdtemp(prefix="pyarrow-"))
(base / "json_dataset").mkdir(exist_ok=True)

# Make file 1
f = open(base / "json_dataset/data1.json", "a")
f.write('{"userId": null}\n{"userId": null}')
f.close()

# Make file 2
f = open(base / "json_dataset/data2.json", "a")
f.write('{"userId": null}\n{"userId": "example-string"}')
f.close()

# writing it into two parquet files
table = ds.dataset(base / "json_dataset", format="json")
print(table.to_table())
```

In this repro, if you change the order of the files, it'll work.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.