julien-duponchelle / julien-duponchelle/python-mysql-replication

Missing events from new databases (MySQL 8)

Open
#405 14 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.4k
Forks
690
PR merge metrics
No merged PRs in 30d

Description

Hi there,
After upgrading my MySQL server from 5.7 to 8.0, I had a problem that events from newly created databases do not appear in the data read by the binlog reader. I used this simple code to test this and found that the data does not appear in the output if the database (table?) is created when this code is already running. And they does appear after if the code is restarted, but only the ones that occur after a restart.

```
import config as cfg
import logging
from pymysqlreplication import BinLogStreamReader
from pymysqlreplication.row_event import (
DeleteRowsEvent,
UpdateRowsEvent,
WriteRowsEvent,
)

while True:
try:
stream = BinLogStreamReader(
connection_settings={'host': cfg.client.host, 'user': cfg.client.user, 'passwd': cfg.client.password},
server_id=11111, blocking=True, resume_stream=True, only_events=[DeleteRowsEvent, WriteRowsEvent, UpdateRowsEvent])
for event in stream:
if event.table != 'posts':
continue
for row in event.rows:
print(row)
except Exception as e:
logging.exception(e)
```

MySQL server is AWS RDS instance, binlog/gtid settings are below:
```
log_bin = ON
binlog_format = ROW
binlog_row_image = FULL
sync_binlog = 0
...
enforce_gtid_consistency = OFF
gtid_mode = OFF_PERMISSIVE
```

mysql-replication is also upgraded to latest version

Contributor guide

Open the contributing guide

Research direction

Start with the BinLogStreamReader example in the issue and reproduce it against the reported MySQL 8 AWS RDS configuration. Compare events for databases created while the stream is running with events after restarting the reader. Done means events from newly created databases are received without restarting the reader.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.