[Umbrella] Spark Datasource Support

Yet to be fully scoped out

But high level, we want to 
* First class support for streaming reads/writes via structured streaming
* Row based reader/writers all the way
* Support for File/Partition pruning using Hudi metadata tables

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1297
- Type: Epic
- Affects version(s):
- 0.9.0

---

## Comments

28/Sep/20 01:50;309637554;[~vinoth] 

very agree with you.

hudi need a sql layer, now presto just can read, spark is  a good choice. Also I think deltastreamer  tool now is useful. But user  with large amount of data will like spark steaming or sql. I think three thing can do .
# ease of use data source, support like binlog (include all table's  binlog of a mysql instance) easy sync to hudi, use spark dataset or sql api （maybe combination of  merge、 upsert、delete）
# ease of use batch transform, and streaming etl. spark & structured streaming  support easy source and sink datasource
# performance support ,  File/Partition pruning using Hudi metadata tables, also with version of hudi can do some cache optimize

but first we can upgrade hudi to spark 3.0 

 ;;;

---

02/Oct/20 21:36;vinoth;yes. thats happening as we speak. support for 3.0;;;

Start with the linked HUDI-1297 epic and scope the requested Spark Structured Streaming reads and writes, row-based readers and writers, and Hudi metadata-table file and partition pruning. No files, tests, or entry points are named, so the first milestone is a decided implementation plan and acceptance criteria for each capability.

Written by the indexing model from the issue text.