authorjapps / authorjapps/zerocode
Set up Kafka cluster in IBM Cloud using Kubernetes and Docker
- Dominant language
- Java
- Stars
- 1k
- Forks
- 453
- Avg merge
- 7d 2h
- Merged PRs (30d)
- 5
Description
### Objective:
+ Setup a data pipeline on IBM Cloud
+ Start from single node, then scale to multi node
+ Source system - PostGres existing(legacy data)
+ Target System - PostGres(or MySql) empty DB
+ Then switch to HBase/Hadoop via another Topic(CDC)
+ Read here [CDC to Kafka for Real-Time Data Integration](https://www.striim.com/cdc-to-kafka/)
+ Read here [Integrating Databases and Kafka with Change Data Capture](https://www.oreilly.com/learning/making-sense-of-stream-processing/page/3/integrating-databases-and-kafka-with-change-data-capture)
+ Use Kafka Connect for the Pipe-Line.
### Why IBM Cloud?
+ For time being, we can use up some free spaces which IBM Cloud offers.
Once the setup is stable, we can go for a subscription
@rpayal to expand this section
### Why Kubernetes?
+ Helps in orchestrating the containerised deployments
@rpayal to expand this section
Contributor guide
Assessment
This issue has not been assessed yet.