[UMBRELLA] Support schema inference for unstructured data

(More details to be added)

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1390
- Type: Epic

---

## Comments

14/Nov/20 16:22;309637554;hello [~xushiyan] now i also do something about this in my work. just as inference unstructured data as csv、json、xml etc.  What about this issue background?  we can have more discussion:D;;;

---

14/Nov/20 20:35;xushiyan;[~309637554] glad to hear that! we labelled a series of issues with gsoc2021 for Google Summer of Code next year (basically a 3-month internship program). The plan was to prepare some non-urgent issues for this program. This issue is exactly what you described. So what do you have in-mind on the timeline to work on this? [~vinoth] maybe we can start the work for this earlier?;;;

---

15/Nov/20 10:32;309637554;[~xushiyan] it is a good news. i see this issue is of bootstrap component. It means that json, csv, xml formats can be directly managed by hudi use bootstrap method ?;;;

---

15/Nov/20 19:14;xushiyan;yes [~309637554] that is the intended use case. It'll be great to be able to import more datasets smoothly into Hudi via bootstrap feature, to get a wider adoption.;;;

---

16/Nov/20 01:19;vinoth;[~309637554] I think we can still start a RFC for this, outline a solution and the intern could help make it better, redesign and take over the RFC when there.  What I am saying is - while we want to keep this work for gsoc interns, it does not prevent us from fleshing it out more. In fact, it may help the intern ramp up quickly and create more impact. WDYT? ;;;

---

16/Nov/20 16:57;xushiyan;yes sounds good.;;;

---

17/Nov/20 01:49;309637554;[~vinoth] [~xushiyan]

1、 Now i am  doing something infer the schema and partitions of  files in object storage such as s3 or oss of aliyun.  Then create table in catalog to store the columns and partition,  just like crawler of glue. The files type contains csv、json、parquet、orc etc. Then the spark or presto can analytics the data use catalog.

2、I think hudi provide a more powerful data manager ability like commit、metadata、index. As the scenario of 1. If we can use hudi manage the unstructured and structured data in object storage without moving data, and also support the update、commit ability of hudi to the data. It will be valuable .

3. i am familiar with the bootstrap code, and  Will do some research. Then start a RFC , may be need some times :);;;

---

20/Nov/20 17:42;vinoth;Awesome!  That would be great.;;;

Start with the Hudi bootstrap code and the linked JIRA issue, then review the proposed RFC discussion about inferring schemas and partitions from object-storage files. The issue is ready only when the RFC defines the supported formats, scope, and acceptance criteria for managing unstructured data through bootstrap.

Written by the indexing model from the issue text.