apache / apache/arrow

[Python] Allow parquet::WriterProperties::created_by to be set via pyarrow.ParquetWriter for compatibility with older parquet-mr

Open
#29,986 16 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

have a couple of files  and using  pyarrow.table (0.17)
to save it as parquet on disk (parquet version 1.4)

colums
id : string
val : string

**table = pa.Table.from_pandas(df)**
**pq.write_table(table, "df.parquet", version='1.0', flavor='spark', write_statistics=True, )**

However, Hive and Spark does not recognize the parquet version:

`org.apache.parquet.VersionParser$VersionParseException: Could not parse created_by: parquet-cpp version 1.5.1-SNAPSHOT using format: (.+) version ((.*) )?(build ?(.*))`
\` at org.apache.parquet.VersionParser.parse(VersionParser.java:112)`
\` at org.apache.parquet.CorruptStatistics.shouldIgnoreStatistics(CorruptStatistics.java:60)`
\` at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:263)`

 

**It seems related to this issue:**

It appears you've encountered PARQUET-349 which was fixed in 2015 before Arrow was even started. The underlying C++ code does allow this `created_by` field to be customized [source](https://github.com/apache/arrow/blob/4591d76fce2846a29dac33bf01e9ba0337b118e9/cpp/src/parquet/properties.h#L249) but the python wrapper does not expose this [source](https://github.com/apache/arrow/blob/4591d76fce2846a29dac33bf01e9ba0337b118e9/python/pyarrow/_parquet.pxd#L360). 

  

**EDIT Add infos**

Current python wrapper does NOT expose :  created_by builder  (when writing parquet on disk)

 

But, this is available in CPP version:

 

This creates an issue when Hadoop parquet reader reads this pyarrow parquet file:

 

 

**SO Question here:**

 

**Reporter**: [Kevin](https://issues.apache.org/jira/browse/ARROW-14422)
#### Related issues:
- [VersionParser does not handle versions like "parquet-mr 1.6.0rc4"](https://issues.apache.org/jira/browse/PARQUET-349) (is related to)

**Note**: *This issue was originally created as [ARROW-14422](https://issues.apache.org/jira/browse/ARROW-14422). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Read cpp/src/parquet/properties.h and python/pyarrow/_parquet.pxd, then trace the pyarrow.ParquetWriter entry point to compare the exposed Python options with the C++ created_by builder. Done means created_by can be configured through ParquetWriter and the resulting metadata supports compatibility with older parquet-mr readers.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.