julien-duponchelle / julien-duponchelle/python-mysql-replication
Missing events from new databases (MySQL 8)
- Dominant language
- Python
- Stars
- 2.4k
- Forks
- 690
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
After upgrading my MySQL server from 5.7 to 8.0, I had a problem that events from newly created databases do not appear in the data read by the binlog reader. I used this simple code to test this and found that the data does not appear in the output if the database (table?) is created when this code is already running. And they does appear after if the code is restarted, but only the ones that occur after a restart.
```
import config as cfg
import logging
from pymysqlreplication import BinLogStreamReader
from pymysqlreplication.row_event import (
DeleteRowsEvent,
UpdateRowsEvent,
WriteRowsEvent,
)
while True:
try:
stream = BinLogStreamReader(
connection_settings={'host': cfg.client.host, 'user': cfg.client.user, 'passwd': cfg.client.password},
server_id=11111, blocking=True, resume_stream=True, only_events=[DeleteRowsEvent, WriteRowsEvent, UpdateRowsEvent])
for event in stream:
if event.table != 'posts':
continue
for row in event.rows:
print(row)
except Exception as e:
logging.exception(e)
```
MySQL server is AWS RDS instance, binlog/gtid settings are below:
```
log_bin = ON
binlog_format = ROW
binlog_row_image = FULL
sync_binlog = 0
...
enforce_gtid_consistency = OFF
gtid_mode = OFF_PERMISSIVE
```
mysql-replication is also upgraded to latest version
Contributor guide
Research direction
Start with the BinLogStreamReader example in the issue and reproduce it against the reported MySQL 8 AWS RDS configuration. Compare events for databases created while the stream is running with events after restarting the reader. Done means events from newly created databases are received without restarting the reader.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100