apache / apache/pulsar

[Bug] Issue in using relational db mysql and postgres as source for pulsar io topics

Open
#22,083 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

Ubuntu 18.04
Pulsar 3.0.1
Mysql - Server version: 5.7.25-log MySQL Community Server (GPL)
Postgres - 14.10

### Minimal reproduce step

Step 1. Follow the steps documented for bringing CDC connectors outlined in example
https://pulsar.apache.org/docs/3.0.x/io-cdc-debezium/
Docker run mysql instance
Step 2. Start a standalone pulsar instance
bin/pulsar standalone
Step 3. Start the Pulsar Debezium connector in local run mode
Attached yaml file (debezium connector config) and logs of execution
Step 4. No topics created for any table from "inventory" schema
Only offset and history topics are created in
[pulsar-debezium-mysql.log](https://github.com/apache/pulsar/files/14351115/pulsar-debezium-mysql.log)
pulsar
[debezium-mysql-source-config.txt](https://github.com/apache/pulsar/files/14351137/debezium-mysql-source-config.txt)

Above was repeated for postgres and same results.
**However if using pulsar 2.6.0 all topics and tables are created successfully.**

### What did you expect to see?

No topic created for any table of the mysql and postgres in pulsar 3.0.1.
Only following topics created for mysql table-
persistent://public/default/offset-topic
persistent://public/default/history-topic

### What did you see instead?

Subscribe to the topic sub-products for the inventory.products table and get change data from relational db.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the 3.0.x CDC connector documentation and the attached debezium-mysql-source-config.txt and pulsar-debezium-mysql.log files. Reproduce the standalone run with MySQL and PostgreSQL, then compare it with Pulsar 2.6.0. Done means table topics are created and emit relational database change data rather than only offset and history topics.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, mysql, postgresql
Domain
databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.