apache / apache/hudi

Support dynamic schemas with hudi

Open
#14,817 2 comments 0 reactions 1 assignee Claimed by @nsivabalan View on GitHub
area:core from-jira priority:high status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Sometimes, users have a requirement where they have different producers and each producer produces only a subset of columns. 

 

for eg:

Producer 1: rec_key, colA, colB, colC

Producer 2: rec_key, colC, colD, colE, colF

Producer 3: rec_key, colB, colF, colI, colK

 

Expectation from hudi:

keep merging new columns and inject defaults values for all other missing columns. 

 

So, for above usecase, final hudi table's schema is expected to be 

rec_key, colA, colB, colC, colD, colE, colF, colI, colK

 

 

 

 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-2175
- Type: Improvement

---

## Comments

15/Mar/22 01:24;x1q1j1;hi [~shivnarayan] I think this implementation is compatible with query, but it is not good enough. This scenario is very common in machine learning and feature engineering. Several features (data columns) are calculated each time through the machine learning algorithm.

I think we should avoid loading all the data when reading the required columns and then filtering. We should support column storage first. For example, we need to add column family like HBase, write separate data files according to the columns when writing data, and read according to the columns when reading.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.