[Python] PyArrow: RuntimeError: AppendRowGroups requires equal schemas when writing _metadata file
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
I'm trying to follow the example here: to write an example partitioned dataset. But I'm consistently getting an error about non-equal schemas. Here's a mcve:
```java
from pathlib import Path
import numpy as np
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
size = 100_000_000
partition_col = np.random.randint(0, 10, size)
values = np.random.rand(size)
table = pa.Table.from_pandas(
pd.DataFrame({"partition_col": partition_col, "values": values})
)
metadata_collector = []
root_path = Path("random.parquet")
pq.write_to_dataset(
table,
root_path,
partition_cols=["partition_col"],
metadata_collector=metadata_collector,
)
Write the ``_common_metadata`` parquet file without row groups statistics
pq.write_metadata(table.schema, root_path / "_common_metadata")
Write the ``_metadata`` parquet file with row groups statistics of all files
pq.write_metadata(
table.schema, root_path / "_metadata", metadata_collector=metadata_collector
)
```
This raises the error
```java
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Input In [92], in ()
----> 1 pq.write_metadata(
2 table.schema, root_path / "_metadata", metadata_collector=metadata_collector
3 )
File ~/tmp/env/lib/python3.8/site-packages/pyarrow/parquet.py:2324, in write_metadata(schema, where, metadata_collector, **kwargs)
2322 metadata = read_metadata(where)
2323 for m in metadata_collector:
-> 2324 metadata.append_row_groups(m)
2325 metadata.write_metadata_file(where)
File ~/tmp/env/lib/python3.8/site-packages/pyarrow/_parquet.pyx:628, in pyarrow._parquet.FileMetaData.append_row_groups()
RuntimeError: AppendRowGroups requires equal schemas.
```
But all schemas in the `metadata_collector` list seem to be the same:
```java
all(metadata_collector[0].schema == meta.schema for meta in metadata_collector)
# True
```
**Environment**: MacOS. Python 3.8.10.
pyarrow: '7.0.0'
pandas: '1.4.2'
numpy: '1.22.3'
**Reporter**: [Kyle Barron](https://issues.apache.org/jira/browse/ARROW-16287)
**Note**: *This issue was originally created as [ARROW-16287](https://issues.apache.org/jira/browse/ARROW-16287). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by running the minimal example from the issue and inspecting the pq.write_metadata call that appends metadata_collector entries. Compare the schemas involved in the generated partition files and the table schema; done means the documented example writes _common_metadata and _metadata without the equal-schema RuntimeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100