apache / apache/hudi

[SUPPORT] Support MetaSync Even there is no new data, Just create the table if not exist in DeltaStream

Open
#8,509 2 comments 0 reactions 0 assignees View on GitHub
component:catalog-sync priority:medium type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I have wanted to sync the hudi table in HIVE Catalog, but not able to sync, due to no new data in Source while using DeltaStream.
It should create a table if not exist or leave as it. if it is empty otherwise do what it was doing earlier.

**Environment Description**

* Hudi version :0.12.1

* Spark version : 3.3.0

* Storage (HDFS/S3/GCS..) : s3

* Running on Docker? (yes/no) : no

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the DeltaStream MetaSync path that handles an empty source, using the Hudi 0.12.1 and Spark 3.3.0 environment described in the issue. Verify the behavior for a missing Hive Catalog table, an existing table, and a source with new data; done means the missing table is created or the existing table is left unchanged when there is no new data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.