[Python] pyarrow.json.read_json ignores nullable=False in explicit_schema parse_options
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Reproducible example:
```python
import json
import pyarrow.json as pj
import pyarrow as pa
s = {"id": "value", "nested": {"value": 1}}
with open("issue.json", "w") as write_file:
json.dump(s, write_file, indent=4)
schema = pa.schema([
pa.field("id", pa.string(), nullable=False),
pa.field("nested", pa.struct([pa.field("value", pa.int64(), nullable=False)]))
])
table = pj.read_json('issue.json', parse_options=pj.ParseOptions(explicit_schema=schema))
print(schema)
# id: string not null
# nested: struct
# child 0, value: int64 not null
print(table.schema)
# id: string
# nested: struct
# child 0, value: int64
```
**Reporter**: [Alenka Frim](https://issues.apache.org/jira/browse/ARROW-16603) / @AlenkaF
**Note**: *This issue was originally created as [ARROW-16603](https://issues.apache.org/jira/browse/ARROW-16603). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Run the reproducible example using issue.json, pyarrow.json.read_json, and ParseOptions(explicit_schema=schema) to confirm the schema mismatch. Trace the read_json explicit-schema path and determine where nullable=False is lost, then verify that the returned table preserves the non-nullable id and nested.value fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100