[Python][Parquet] Pyarrow seems to drop logical_type and converted_type in pyarrow.parquet.write_metadata leading to RuntimeError: AppendRowGroups requires equal schemas. error
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I'm trying to write a `_metadata` file from a set of existing parquet files written by duckdb.
This is the minimal example I could come up with that reproduces the issue
```python
import duckdb
import pyarrow.parquet as pq
fname = 'parquet_file.parquet'
con = duckdb.connect()
con.sql(("CREATE TABLE t1 (num INTEGER);"
"INSERT INTO t1 VALUES (1);"
"INSERT INTO t1 VALUES (2);"))
con.sql(f"COPY t1 TO '{fname}' (FORMAT PARQUET)")
metadata = pq.read_metadata(fname)
schema = pq.read_schema(fname)
metadata_path = "_metadata"
pq.write_metadata(
schema, metadata_path,
metadata_collector=[metadata, metadata])
```
leads to
```
Traceback (most recent call last):
File "/Users/christoph/debugging/2024-09-27-parquet-speed/parquet-tests/pyarrow_error.py", line 17, in
pq.write_metadata(
File "/Users/christoph/debugging/2024-09-27-parquet-speed/parquet-tests/.venv/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 2247, in write_metadata
metadata.append_row_groups(m)
File "pyarrow/_parquet.pyx", line 1047, in pyarrow._parquet.FileMetaData.append_row_groups
RuntimeError: AppendRowGroups requires equal schemas.
The two columns with index 0 differ.
column descriptor = {
name: num,
path: num,
physical_type: INT32,
converted_type: NONE,
logical_type: None,
max_definition_level: 1,
max_repetition_level: 0,
}
column descriptor = {
name: num,
path: num,
physical_type: INT32,
converted_type: INT_32,
logical_type: Int(bitWidth=32, isSigned=true),
max_definition_level: 1,
max_repetition_level: 0,
}
```
Package versions
```
"duckdb==1.1.2",
"pyarrow>=17.0.0"
```
### Component(s)
Parquet
Contributor guide
Assessment
This issue has not been assessed yet.