apache / apache/arrow

[Python] Schema inference reorders fields in nested structs

Open
#34,250 2 comments 2 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Code example:
```
>>> import pyarrow
>>> pyarrow.RecordBatch.from_pylist([{"start": 0, "end": 1, "tag": "foo"}]).schema
start: int64
end: int64
tag: string
>>> pyarrow.RecordBatch.from_pylist([{"spans": [{"start": 0, "end": 1, "tag": "foo"}]}]).schema
spans: list>
child 0, item: struct
child 0, end: int64
child 1, start: int64
child 2, tag: string
```

In the 1st schema, the fields of the `struct` are in the same order as the keys in the input dictionary. In the 2nd schema, where the `struct` is nested inside a `list`, the fields of the `struct` have been sorted by name. I would expect the ordering to always be the same order as in the input (like in the 1st schema). The more general principle would be that the input -- or at least the first row of input that's used for schema inference -- should validate against the inferred schema.

I suspect this behavior is related to `from_pylist()` accepting lists where the elements are dictionaries with different key sets, such as:
```
>>> pyarrow.RecordBatch.from_pylist([{"spans": [{"start": 0, "end": 1, "tag": "foo"}, {"new": 42}]}]).schema
spans: list>
child 0, item: struct
child 0, end: int64
child 1, new: int64
child 2, start: int64
child 3, tag: string
```
In this case I'm guessing it sorts the fields because it needs to come up with a canonical ordering. It seems to me that it should just fail to infer a schema here, since again the inputs are not valid according to the inferred schema.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the nested examples with pyarrow.RecordBatch.from_pylist and inspect the schema-inference path used for lists of dictionaries. Compare top-level and nested struct field ordering, then add or update coverage so the inferred schema preserves input order or explicitly rejects incompatible key sets.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.