Implement DeltaStreamer EventsSource and EventsHoodieIncrSource for Azure Blob storage
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
For Azure blob storage, similar to HUDI-1897
GH issue request
- [https://github.com/apache/hudi/issues/7158]
Since Streaming ingestion using DeltaStreamer from DFS that contains parquet files has a problem as explained in this blog here [https://medium.com/apache-hudi-blogs/reliable-ingestion-from-aws-s3-using-hudi-b7d5590c78a9] here, I'm working on implementing a similar setup described in the above mentioned blog for ingesting parquet files stored in Azure blob storage and enable event triggers to Azure storage queue.
Currently, hudi-utilities/sources contains support only for S3 events source (S3EventsSource.java) and incremental pulls from S3 (S3EventsHoodieIncrSource.java). The ingestion pattern doesn't seem to support equivalent Azure cloud stack.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5202
- Type: New Feature
- Epic: https://issues.apache.org/jira/browse/HUDI-1896
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.