apache / apache/arrow

[Python] pyarrow.json.read_json ignores nullable=False in explicit_schema parse_options

Open
#31,957 1 comment 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Reproducible example:
```python

import json
import pyarrow.json as pj
import pyarrow as pa

s = {"id": "value", "nested": {"value": 1}}

with open("issue.json", "w") as write_file:
json.dump(s, write_file, indent=4)

schema = pa.schema([
pa.field("id", pa.string(), nullable=False),
pa.field("nested", pa.struct([pa.field("value", pa.int64(), nullable=False)]))
])

table = pj.read_json('issue.json', parse_options=pj.ParseOptions(explicit_schema=schema))

print(schema)
# id: string not null
# nested: struct
#  child 0, value: int64 not null
print(table.schema)
# id: string
# nested: struct
#   child 0, value: int64
```

**Reporter**: [Alenka Frim](https://issues.apache.org/jira/browse/ARROW-16603) / @AlenkaF

**Note**: *This issue was originally created as [ARROW-16603](https://issues.apache.org/jira/browse/ARROW-16603). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Run the reproducible example using issue.json, pyarrow.json.read_json, and ParseOptions(explicit_schema=schema) to confirm the schema mismatch. Trace the read_json explicit-schema path and determine where nullable=False is lost, then verify that the returned table preserves the non-nullable id and nested.value fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.