parquet-cascading Add Parquet Avro Scheme
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
At the moment using parquet-cascading writing Parquet files when using Avro as the object format is not possible as a Schema is not yet provided.
As part of this ticket, a ParquetAvroScheme will be introduced, and also a local.ParquetAvroScheme , so that from Cascading / Scalding developers can read / write and test in both --hdfs and --local mode Parquet-Avro files.
**Reporter**: [Antonios Chalkiopoulos](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=antwnis) / @Antwnis
**Assignee**: [Antonios Chalkiopoulos](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=antwnis) / @Antwnis
**Note**: *This issue was originally created as [PARQUET-198](https://issues.apache.org/jira/browse/PARQUET-198). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the parquet-cascading Scheme implementations and the local scheme counterparts, then identify how Cascading and Scalding handle Avro object formats. Add the requested ParquetAvroScheme and local.ParquetAvroScheme, with coverage for reading, writing, and testing Parquet-Avro files in HDFS and local modes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100