baidu / baidu/bigflow

Support structure IO format on Spark

Open
#11 1 comment 0 reactions 2 assignees Claimed by @chunyang-wen View on GitHub
enhancement
Dominant language
C++
Stars
1.1k
Forks
159
PR merge metrics
No merged PRs in 30d

Description

## Definitions
Structure input formats specifically mean [ORC](https://github.com/apache/orc) file and [Parquet](https://github.com/apache/parquet-format) file.

## Current Status
Bigflow on DCE supports ORC file(only reading) and Parquet file with its own loader as DCE doesn't support reading ORC or Parquet natively.

For ORC files, Bigflow uses ORC's c++ [API](https://github.com/apache/orc/tree/master/c%2B%2B). As the time of adding ORC support, ORC's c++ API only supports reading.

For Parquet files, Bigflow also uses c++ [API](https://github.com/apache/parquet-cpp). Currently, parquet-cpp partially supports nested structure.

Bigflow on Spark doesn't support ORC neither Parquet for now. This doc lists some details how we can support for ORC and Parquet files.

## Parquet Support Architecture Overview on DCE
![parquet_architecture](https://user-images.githubusercontent.com/807537/33277093-7965fb9a-d3d2-11e7-8175-63442a94988b.png)

ORC loader follows similar procedure.

## How to add support for spark pipeline

### Read support
The RecordBatch in the previous arch is an [arrow](https://github.com/apache/arrow)
RecordBatch. Spark already adds supports to transform `Dataset` to `RDD[ArrowPayload]`
(see [Dataset.scala](https://github.com/apache/spark/blob/154351e6dbd24c4254094477e3f7defcba979b1a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L3207)), though not publicly.

It would be straightforward to add Parquet read support on spark pipeline, even ORC or CSV files.

### Impl details to add read support
1. Use SparkSession to read Parquet or Orc File(spark pipeline currently uses SparkContext)
2. Implements `toArrowPayload` in flume-rumtime as Spark doesn't expose that publicly
3. Reuse and refactoring current `PythonFromRecordBatchProcessor`
4. Modify Bigflow's planner to use PythonFromRecordBatchProcessor for Spark pipeline's structure input when constructing Flume task

### Write support
Bigflow uses its own sinker impl to write PCollection(or PType) into external target.

Current impl on DCE should also works on Spark. Although, some additional work is
needed, namely:

1. Refactoring current ParquetSinker and Arrow Schema Converter
2. Add write support for ORC files. (ORC's cpp API is adding write support incrementally)

## References
1. Apache Arrow is a promising in-memory columnar storage, we can leverage more
power on it. See [Arrow SlideShare](https://www.slideshare.net/julienledem/improving-python-and-spark-performance-and-interoperability-with-apache-arrow)

cc @himdd @chunyang-wen @bb7133 @acmol for comments and prs are appreciated

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.