apache / apache/gravitino

[Improvement] How to get the schema from the fileset data

Open
#3,407 0 comments 1 reaction 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 15h
Merged PRs (30d)
315

Description

### What would you like to be improved?

For the fileset data, there always a lot of files and gather together. How to check and extract the schema is a very important feature, and it is very useful in the data engineer's daily work.
1. Provide the path of the firesets, go through each file, extract the schema
2. If the schema are same, store the schema, path and related information somewhere.
3. If the schema are different, create several schema. For each schema, it has its own path and other related information.
4. The schema is a combination of the key and value pair. The key is the column name, the value is the column type.
5. If the column type is an array or an object(json/dict), store the sub structure along with this column name and column type.

### How should we improve?
We can check how Crawler work in AWS, it will help us to understand the schema extracting process a lot.

There is a [video](https://www.youtube.com/watch?v=p4PiMyNZvks) to show how crawler works.

Below is how the schema looks like after the crawler extract the schema from fileset data.
![截屏2024-05-15 下午3 24 23](https://github.com/datastrato/gravitino/assets/141538510/394aa794-7120-470f-9e74-8577f6fa04e8)

if the data type is struct(json/dict), then we can store the structure of the value, check the example as blew
截屏2024-05-15 下午6 18 25

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the repository's existing fileset and schema-related entry points, then compare the requested behavior with the AWS Crawler example and linked video. Define how paths, shared schemas, differing schemas, column types, and nested JSON or dictionary structures should be represented. Done means fileset data can be scanned and each resulting schema is associated with its paths and related information.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.