julien-duponchelle / julien-duponchelle/python-mysql-replication
Missing Data
- Dominant language
- Python
- Stars
- 2.4k
- Forks
- 690
- PR merge metrics
- No merged PRs in 30d
Description
Dear Developers/community
I am using mysql replication library to stream my aws RDS binlogs to redshift. Everything is working fine except one thing, I am not getting some records from a table.
After investigating, I came to know that data in my table is being populated though a script and this script is inserting multiple rows in one insert statement.
Please note that I am reading binlogs of a slave server. This is replica of my production. Point here is changes of this table are propagating to all slaves perfectly. I am not sure why python is not reading some of changes.
Here is my pyton code snippet
`stream = BinLogStreamReader(
connection_settings={
"host": SRC_DB_HOST,
"port": src_db_port,
"user": user_bi,
"passwd": user_bi_pwd},
server_id=227336,
blocking=True,
resume_stream=True,
ignored_tables=ignored_tables_list,
log_file=log_file, log_pos=log_pos,
only_events=[WriteRowsEvent, UpdateRowsEvent, DeleteRowsEvent])
for binlogevent in stream:
##doing some stuff here`
ignored_tables_list: this contains some tables to exclude
There are no invalid characters in my missing records. They are same as others.
Please help
Contributor guide
Research direction
Start with the BinLogStreamReader configuration and the WriteRowsEvent handling, focusing on how multi-row INSERT statements from a slave binlog are exposed to the iterator. Reproduce the report with a multi-row insert and verify whether every row is emitted; done means identifying and correcting the omission, with coverage for that scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100