julien-duponchelle / julien-duponchelle/python-mysql-replication

Missing Data

Open
#264 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.4k
Forks
690
PR merge metrics
No merged PRs in 30d

Description

Dear Developers/community

I am using mysql replication library to stream my aws RDS binlogs to redshift. Everything is working fine except one thing, I am not getting some records from a table.
After investigating, I came to know that data in my table is being populated though a script and this script is inserting multiple rows in one insert statement.
Please note that I am reading binlogs of a slave server. This is replica of my production. Point here is changes of this table are propagating to all slaves perfectly. I am not sure why python is not reading some of changes.

Here is my pyton code snippet

`stream = BinLogStreamReader(
connection_settings={
"host": SRC_DB_HOST,
"port": src_db_port,
"user": user_bi,
"passwd": user_bi_pwd},
server_id=227336,
blocking=True,
resume_stream=True,
ignored_tables=ignored_tables_list,
log_file=log_file, log_pos=log_pos,
only_events=[WriteRowsEvent, UpdateRowsEvent, DeleteRowsEvent])

for binlogevent in stream:
##doing some stuff here`

ignored_tables_list: this contains some tables to exclude

There are no invalid characters in my missing records. They are same as others.

Please help

Contributor guide

Open the contributing guide

Research direction

Start with the BinLogStreamReader configuration and the WriteRowsEvent handling, focusing on how multi-row INSERT statements from a slave binlog are exposed to the iterator. Reproduce the report with a multi-row insert and verify whether every row is emitted; done means identifying and correcting the omission, with coverage for that scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.