apache / apache/arrow

Unexpected Behaviour of nullable fields

Open
#36,060 5 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the usage question you have. Please include as many useful details as possible.

The following problem:
If an optional field is not specified in the json, but is in the schema, it is still created in the pyarrow table, including all nested fields that are specified in the schema (with null values).
Is this the intended behaviour or is there a setting option so that non-existent fields are also not in the table?

Here is a data example. our productive data schema is of course much more complex and more nested, but it illustrates what I am doing:

Schema (all fields are nullable):
```
field1: struct
field2: timestamp[ms]
field3: double

```

json file:
```json
{
"field3": 123.4
}
```

Python code handling the data:

```python
read_options = pajson.ReadOptions(block_size=1600000000)

parse_options = pajson.ParseOptions(
explicit_schema=pa_schema,
unexpected_field_behavior="ignore"
)
table = pajson.read_json(
tmp_file_name, read_options=read_options, parse_options=parse_options
)

pq.write_to_dataset(
table=table,
root_path=dataset_path,
basename_template=hashvalue + ".parquet",
existing_data_behavior="overwrite_or_ignore",
schema=pa_schema
)
```

table debug output from evaluation in pycharm:
```
column_names: ['field1', 'field2', 'field3']
columns:
[
-- is_valid:
[
false
]
-- child 0 type: double
[
null
]
-- child 1 type: double
[
null
]
]
[
[
null
]
]
[
[
123.4
]
]

```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start at the Python pajson.read_json call, especially ReadOptions, ParseOptions, and explicit_schema handling, then compare the resulting table schema with pa_schema. Determine whether omitted nullable fields are intentionally materialized and whether an existing option controls this; done should be a confirmed behavior or a narrowly defined change with coverage for the nested-field example.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.