apache / apache/arrow

[Python] Reading JSON with explicit schema is ignoring not null constraint

Open
#13,177 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Hey!

I'm trying to read json using explicit schema as so:
**Input file** (`issue.json`):
```json
{"id": "value", "nested": {"value": 1}}
{"id": "value", "nested": {"value": 1}}
```

**Code:**
```python
import pyarrow.json as pj
import pyarrow as pa

schema = pa.schema([
pa.field("id", pa.string(), nullable=False),
pa.field("nested", pa.struct([pa.field("value", pa.int64(), nullable=False)]))
])

table = pj.read_json('./issue.json', parse_options=pj.ParseOptions(explicit_schema=schema))

print(schema)
print(table.schema)
```

But the table schema is different - it doesn't contain the not null constraint.

**Provided explicit schema:**
```
id: string not null
nested: struct
child 0, value: int64 not null
```

**Table schema:**
```
id: string
nested: struct
child 0, value: int64
```

I was trying also casting the table schema (`table.cast(schema`) and it works for top level not null constraint but for nested struct it throws an error:
```
pyarrow.lib.ArrowTypeError: cannot cast nullable field to non-nullable field: struct struct
```

Is there another way to force the schema?

Contributor guide

Open the contributing guide

Research direction

Start with the Python pyarrow.json.read_json entry point and its ParseOptions(explicit_schema=...) handling, reproducing the behavior with the issue.json example. Trace how nullability is propagated into the returned table, including nested structs; done means explicit top-level and nested not-null constraints are preserved and the reproducer passes without the reported cast error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.