[Python] Reading JSON with explicit schema is ignoring not null constraint
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Hey!
I'm trying to read json using explicit schema as so:
**Input file** (`issue.json`):
```json
{"id": "value", "nested": {"value": 1}}
{"id": "value", "nested": {"value": 1}}
```
**Code:**
```python
import pyarrow.json as pj
import pyarrow as pa
schema = pa.schema([
pa.field("id", pa.string(), nullable=False),
pa.field("nested", pa.struct([pa.field("value", pa.int64(), nullable=False)]))
])
table = pj.read_json('./issue.json', parse_options=pj.ParseOptions(explicit_schema=schema))
print(schema)
print(table.schema)
```
But the table schema is different - it doesn't contain the not null constraint.
**Provided explicit schema:**
```
id: string not null
nested: struct
child 0, value: int64 not null
```
**Table schema:**
```
id: string
nested: struct
child 0, value: int64
```
I was trying also casting the table schema (`table.cast(schema`) and it works for top level not null constraint but for nested struct it throws an error:
```
pyarrow.lib.ArrowTypeError: cannot cast nullable field to non-nullable field: struct struct
```
Is there another way to force the schema?
Contributor guide
Research direction
Start with the Python pyarrow.json.read_json entry point and its ParseOptions(explicit_schema=...) handling, reproducing the behavior with the issue.json example. Trace how nullability is propagated into the returned table, including nested structs; done means explicit top-level and nested not-null constraints are preserved and the reproducer passes without the reported cast error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100