authorjapps / authorjapps/zerocode

Set up Kafka cluster in IBM Cloud using Kubernetes and Docker

Open
#205 0 comments 0 reactions 1 assignee Claimed by @rpayal View on GitHub
kafka
Dominant language
Java
Stars
1k
Forks
453
Avg merge
7d 2h
Merged PRs (30d)
5

Description

### Objective:
+ Setup a data pipeline on IBM Cloud
+ Start from single node, then scale to multi node
+ Source system - PostGres existing(legacy data)
+ Target System - PostGres(or MySql) empty DB
+ Then switch to HBase/Hadoop via another Topic(CDC)
+ Read here [CDC to Kafka for Real-Time Data Integration](https://www.striim.com/cdc-to-kafka/)
+ Read here [Integrating Databases and Kafka with Change Data Capture](https://www.oreilly.com/learning/making-sense-of-stream-processing/page/3/integrating-databases-and-kafka-with-change-data-capture)
+ Use Kafka Connect for the Pipe-Line.

### Why IBM Cloud?
+ For time being, we can use up some free spaces which IBM Cloud offers.
Once the setup is stable, we can go for a subscription
@rpayal to expand this section

### Why Kubernetes?
+ Helps in orchestrating the containerised deployments
@rpayal to expand this section

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.