[Python] Allow parquet::WriterProperties::created_by to be set via pyarrow.ParquetWriter for compatibility with older parquet-mr
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
have a couple of files and using pyarrow.table (0.17)
to save it as parquet on disk (parquet version 1.4)
colums
id : string
val : string
**table = pa.Table.from_pandas(df)**
**pq.write_table(table, "df.parquet", version='1.0', flavor='spark', write_statistics=True, )**
However, Hive and Spark does not recognize the parquet version:
`org.apache.parquet.VersionParser$VersionParseException: Could not parse created_by: parquet-cpp version 1.5.1-SNAPSHOT using format: (.+) version ((.*) )?(build ?(.*))`
\` at org.apache.parquet.VersionParser.parse(VersionParser.java:112)`
\` at org.apache.parquet.CorruptStatistics.shouldIgnoreStatistics(CorruptStatistics.java:60)`
\` at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:263)`
**It seems related to this issue:**
It appears you've encountered PARQUET-349 which was fixed in 2015 before Arrow was even started. The underlying C++ code does allow this `created_by` field to be customized [source](https://github.com/apache/arrow/blob/4591d76fce2846a29dac33bf01e9ba0337b118e9/cpp/src/parquet/properties.h#L249) but the python wrapper does not expose this [source](https://github.com/apache/arrow/blob/4591d76fce2846a29dac33bf01e9ba0337b118e9/python/pyarrow/_parquet.pxd#L360).
**EDIT Add infos**
Current python wrapper does NOT expose : created_by builder (when writing parquet on disk)
But, this is available in CPP version:
This creates an issue when Hadoop parquet reader reads this pyarrow parquet file:
**SO Question here:**
**Reporter**: [Kevin](https://issues.apache.org/jira/browse/ARROW-14422)
#### Related issues:
- [VersionParser does not handle versions like "parquet-mr 1.6.0rc4"](https://issues.apache.org/jira/browse/PARQUET-349) (is related to)
**Note**: *This issue was originally created as [ARROW-14422](https://issues.apache.org/jira/browse/ARROW-14422). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Read cpp/src/parquet/properties.h and python/pyarrow/_parquet.pxd, then trace the pyarrow.ParquetWriter entry point to compare the exposed Python options with the C++ created_by builder. Done means created_by can be configured through ParquetWriter and the resulting metadata supports compatibility with older parquet-mr readers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100