Support structure IO format on Spark
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 159
- PR merge metrics
- No merged PRs in 30d
Description
## Definitions
Structure input formats specifically mean [ORC](https://github.com/apache/orc) file and [Parquet](https://github.com/apache/parquet-format) file.
## Current Status
Bigflow on DCE supports ORC file(only reading) and Parquet file with its own loader as DCE doesn't support reading ORC or Parquet natively.
For ORC files, Bigflow uses ORC's c++ [API](https://github.com/apache/orc/tree/master/c%2B%2B). As the time of adding ORC support, ORC's c++ API only supports reading.
For Parquet files, Bigflow also uses c++ [API](https://github.com/apache/parquet-cpp). Currently, parquet-cpp partially supports nested structure.
Bigflow on Spark doesn't support ORC neither Parquet for now. This doc lists some details how we can support for ORC and Parquet files.
## Parquet Support Architecture Overview on DCE

ORC loader follows similar procedure.
## How to add support for spark pipeline
### Read support
The RecordBatch in the previous arch is an [arrow](https://github.com/apache/arrow)
RecordBatch. Spark already adds supports to transform `Dataset` to `RDD[ArrowPayload]`
(see [Dataset.scala](https://github.com/apache/spark/blob/154351e6dbd24c4254094477e3f7defcba979b1a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L3207)), though not publicly.
It would be straightforward to add Parquet read support on spark pipeline, even ORC or CSV files.
### Impl details to add read support
1. Use SparkSession to read Parquet or Orc File(spark pipeline currently uses SparkContext)
2. Implements `toArrowPayload` in flume-rumtime as Spark doesn't expose that publicly
3. Reuse and refactoring current `PythonFromRecordBatchProcessor`
4. Modify Bigflow's planner to use PythonFromRecordBatchProcessor for Spark pipeline's structure input when constructing Flume task
### Write support
Bigflow uses its own sinker impl to write PCollection(or PType) into external target.
Current impl on DCE should also works on Spark. Although, some additional work is
needed, namely:
1. Refactoring current ParquetSinker and Arrow Schema Converter
2. Add write support for ORC files. (ORC's cpp API is adding write support incrementally)
## References
1. Apache Arrow is a promising in-memory columnar storage, we can leverage more
power on it. See [Arrow SlideShare](https://www.slideshare.net/julienledem/improving-python-and-spark-performance-and-interoperability-with-apache-arrow)
cc @himdd @chunyang-wen @bb7133 @acmol for comments and prs are appreciated
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.