apache / apache/arrow

[Python] read_json error message not clear when duplicate field in schema

Open
#44,949 9 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I had mistakenly put a duplicate field in my schema when calling `pyarrow.json.read_json` with `ParseOptions.explicit_schema`

I was either getting seg faults or an error message that wasn't clear.

Given the nature of json, there shouldn't be duplicate fields in the schema. `pyarrow.json.ParseOptions` should throw if there's any duplicate field in the `explicit_schema`.

I can send an MR, I'm just wondering if there's anywhere else where we do similar checks?

```python
import io
import pyarrow as pa
import pyarrow.json

SCHEMA = pa.schema(
[
pa.field("foo", pa.bool_()),
pa.field("foo", pa.bool_()),
]
)

with io.BytesIO(b'{"foo": true,"other": "bar"}') as buffer:
buffer.seek(0)
pyarrow.json.read_json(
buffer,
parse_options=pyarrow.json.ParseOptions(explicit_schema=SCHEMA),
)

```

```
pyarrow.json.read_json(
File "pyarrow/_json.pyx", line 308, in pyarrow._json.read_json
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Failed to convert JSON to bool from dictionary

```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.