Unexpected Behaviour of nullable fields
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the usage question you have. Please include as many useful details as possible.
The following problem:
If an optional field is not specified in the json, but is in the schema, it is still created in the pyarrow table, including all nested fields that are specified in the schema (with null values).
Is this the intended behaviour or is there a setting option so that non-existent fields are also not in the table?
Here is a data example. our productive data schema is of course much more complex and more nested, but it illustrates what I am doing:
Schema (all fields are nullable):
```
field1: struct
field2: timestamp[ms]
field3: double
```
json file:
```json
{
"field3": 123.4
}
```
Python code handling the data:
```python
read_options = pajson.ReadOptions(block_size=1600000000)
parse_options = pajson.ParseOptions(
explicit_schema=pa_schema,
unexpected_field_behavior="ignore"
)
table = pajson.read_json(
tmp_file_name, read_options=read_options, parse_options=parse_options
)
pq.write_to_dataset(
table=table,
root_path=dataset_path,
basename_template=hashvalue + ".parquet",
existing_data_behavior="overwrite_or_ignore",
schema=pa_schema
)
```
table debug output from evaluation in pycharm:
```
column_names: ['field1', 'field2', 'field3']
columns:
[
-- is_valid:
[
false
]
-- child 0 type: double
[
null
]
-- child 1 type: double
[
null
]
]
[
[
null
]
]
[
[
123.4
]
]
```
### Component(s)
Python
Contributor guide
Research direction
Start at the Python pajson.read_json call, especially ReadOptions, ParseOptions, and explicit_schema handling, then compare the resulting table schema with pa_schema. Determine whether omitted nullable fields are intentionally materialized and whether an existing option controls this; done should be a confirmed behavior or a narrowly defined change with coverage for the nested-field example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100