[SUPPORT] Support MetaSync Even there is no new data, Just create the table if not exist in DeltaStream
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I have wanted to sync the hudi table in HIVE Catalog, but not able to sync, due to no new data in Source while using DeltaStream.
It should create a table if not exist or leave as it. if it is empty otherwise do what it was doing earlier.
**Environment Description**
* Hudi version :0.12.1
* Spark version : 3.3.0
* Storage (HDFS/S3/GCS..) : s3
* Running on Docker? (yes/no) : no
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the DeltaStream MetaSync path that handles an empty source, using the Hudi 0.12.1 and Spark 3.3.0 environment described in the issue. Verify the behavior for a missing Hive Catalog table, an existing table, and a source with new data; done means the missing table is created or the existing table is left unchanged when there is no new data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100