apache / apache/arrow

Dataset.to_arrow() can generate invalid tables

Open
#39,670 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Imagine we have a couple of Parquet files with non-nullable columns with different names:
```
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

x = pa.Table.from_arrays([pa.array([1, 2, 3])], schema=pa.schema([pa.field("x", pa.int32(), nullable=False)]))
y = pa.Table.from_arrays([pa.array([1, 2, 3])], schema=pa.schema([pa.field("y", pa.int32(), nullable=False)]))

pq.write_table(x, "x.parquet")
pq.write_table(y, "y.parquet")
```
We can read these with `Dataset` if we provide it with the combined schema:
```
schema = pa.unify_schemas([x.schema, y.schema])
table = ds.dataset(["x.parquet", "y.parquet"], schema=schema).to_table()

print(table)
```
Which gives us a slightly curious result, of a table with non-nullable columns which contain nulls:
```
pyarrow.Table
x: int32 not null
y: int32 not null
----
x: [[1,2,3],[null,null,null]]
y: [[null,null,null],[1,2,3]]
```
Which means that casting a table to its own schema fails (which I don't think should ever happen):
```
>>> table.cast(table.schema)
...
ValueError: Casting field 'x' with null values to non-nullable
```
This also breaks if we don't provide a schema, `Dataset` uses the schema from `x.parquet` so the output is:
```
pyarrow.Table
x: int32 not null
----
x: [[1,2,3],[null,null,null]]
```
So ideally both:
- `Dataset` would notice if it was adding null chunks and fixup the schema of the returned table
- `pa.unify_schemas()` would promote mismatched non-nullable field names to nullable

### Component(s)

C++, Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.