Dataset.to_arrow() can generate invalid tables
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Imagine we have a couple of Parquet files with non-nullable columns with different names:
```
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds
x = pa.Table.from_arrays([pa.array([1, 2, 3])], schema=pa.schema([pa.field("x", pa.int32(), nullable=False)]))
y = pa.Table.from_arrays([pa.array([1, 2, 3])], schema=pa.schema([pa.field("y", pa.int32(), nullable=False)]))
pq.write_table(x, "x.parquet")
pq.write_table(y, "y.parquet")
```
We can read these with `Dataset` if we provide it with the combined schema:
```
schema = pa.unify_schemas([x.schema, y.schema])
table = ds.dataset(["x.parquet", "y.parquet"], schema=schema).to_table()
print(table)
```
Which gives us a slightly curious result, of a table with non-nullable columns which contain nulls:
```
pyarrow.Table
x: int32 not null
y: int32 not null
----
x: [[1,2,3],[null,null,null]]
y: [[null,null,null],[1,2,3]]
```
Which means that casting a table to its own schema fails (which I don't think should ever happen):
```
>>> table.cast(table.schema)
...
ValueError: Casting field 'x' with null values to non-nullable
```
This also breaks if we don't provide a schema, `Dataset` uses the schema from `x.parquet` so the output is:
```
pyarrow.Table
x: int32 not null
----
x: [[1,2,3],[null,null,null]]
```
So ideally both:
- `Dataset` would notice if it was adding null chunks and fixup the schema of the returned table
- `pa.unify_schemas()` would promote mismatched non-nullable field names to nullable
### Component(s)
C++, Parquet, Python
Contributor guide
Assessment
This issue has not been assessed yet.