apache / apache/arrow

[Python][Parquet] Pyarrow seems to drop logical_type and converted_type in pyarrow.parquet.write_metadata leading to RuntimeError: AppendRowGroups requires equal schemas. error

Open
#44,525 1 comment 1 reaction 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I'm trying to write a `_metadata` file from a set of existing parquet files written by duckdb.

This is the minimal example I could come up with that reproduces the issue

```python
import duckdb
import pyarrow.parquet as pq

fname = 'parquet_file.parquet'
con = duckdb.connect()

con.sql(("CREATE TABLE t1 (num INTEGER);"
"INSERT INTO t1 VALUES (1);"
"INSERT INTO t1 VALUES (2);"))
con.sql(f"COPY t1 TO '{fname}' (FORMAT PARQUET)")

metadata = pq.read_metadata(fname)
schema = pq.read_schema(fname)
metadata_path = "_metadata"

pq.write_metadata(
schema, metadata_path,
metadata_collector=[metadata, metadata])
```

leads to

```
Traceback (most recent call last):
File "/Users/christoph/debugging/2024-09-27-parquet-speed/parquet-tests/pyarrow_error.py", line 17, in
pq.write_metadata(
File "/Users/christoph/debugging/2024-09-27-parquet-speed/parquet-tests/.venv/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 2247, in write_metadata
metadata.append_row_groups(m)
File "pyarrow/_parquet.pyx", line 1047, in pyarrow._parquet.FileMetaData.append_row_groups
RuntimeError: AppendRowGroups requires equal schemas.
The two columns with index 0 differ.
column descriptor = {
name: num,
path: num,
physical_type: INT32,
converted_type: NONE,
logical_type: None,
max_definition_level: 1,
max_repetition_level: 0,
}
column descriptor = {
name: num,
path: num,
physical_type: INT32,
converted_type: INT_32,
logical_type: Int(bitWidth=32, isSigned=true),
max_definition_level: 1,
max_repetition_level: 0,
}
```

Package versions

```
"duckdb==1.1.2",
"pyarrow>=17.0.0"
```

### Component(s)

Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.