apache / apache/arrow

Getting a 'pyarrow.lib.ArrowTypeError' error following a schema merge on delta table with nested columns

Open
#43,893 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Following a schema merge operation in a delta table involving nested columns, PyArrow seems to struggle with loading data with the following error:

`pyarrow.lib.ArrowTypeError: struct fields don't match or are in the wrong order: Input fields: struct output fields: struct`

I have confirmed this does not happen with a schema merge that DOES NOT involve any nested columns.

I believe this is a PyArrow specific problem as Spark does not have this problem. My very naive assumption is that PyArrow won't be able to assume that a new column will always be added to the "end" of a schema.

Below is an example of how this can be reproduced:

```
import pyarrow as pa
import polars as pl
from deltalake import write_deltalake

df = pa.table({
"a": [1, 2, 3],
"b": [{"c": 1}, {"c": 2}, {"c": 3}]
})

schema = pa.schema([
pa.field("a", pa.int64()),
pa.field("b", pa.struct([
pa.field("c", pa.int64())
]))
])

local_path = "./tables/merge_delta_table"

# Write the table to delta lake
write_deltalake(local_path, data=df, engine="rust", schema=schema, mode="append")

# Create a new table with a different schema, added a new column 'd' before 'c'.
df2 = pa.table({
"a": [4, 5, 6],
"b": [{"d": 2, "c": 1}, {"c": 2}, {"c": 3}]
})

schema2 = pa.schema([
pa.field("a", pa.int64()),
pa.field("b", pa.struct([
pa.field("d", pa.int64()),
pa.field("c", pa.int64())
]))
])
# Write the new table to the same delta lake
write_deltalake(local_path, data=df2, schema=schema2, engine="rust", mode="append", schema_mode="merge")

# Now read the delta lake using polars
df = pl.read_delta(local_path)
print(df)
```

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.