julien-duponchelle / julien-duponchelle/python-mysql-replication

Critical Bug - Usage of skip_to_timestamp option causes querying the information schema huge number of times

Open
#383 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.4k
Forks
690
PR merge metrics
No merged PRs in 30d

Description

### Overview
We are using the library to periodically sync the updated rows to another database. We are using the `skip_to_timestamp` option to skip the binary logs that were already synced in the previous cycle. But if we use this, we see that the library is executing [this](https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/binlogstream.py#L606-L614) query too many times.

### Bug description
According to the logic written in *row_event.py* ([line 613](https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/row_event.py#L613)), it seems the schema is fetched for every TABLE_MAP_EVENT if it is not already present in the *table_map*. But in binlogstream.py ([lines 551 to 557](https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/binlogstream.py#L551-L557)), the result of the fetched schema is ignored if the event timestamp is lesser than the `skip_to_timestamp` option. So, the schema will be fetched again in the next TABLE_MAP_EVENT as it was not populated in the *table_map* previously.

### Resolution
Populating the *table_map* first before continuing the loop in binlogstream.py ([lines 551 to 557](https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/binlogstream.py#L551-L557)) should fix the issue.

Contributor guide

Open the contributing guide

Research direction

Start by reading the skip_to_timestamp loop in pymysqlreplication/binlogstream.py around lines 551-557, then follow the table-map schema lookup in pymysqlreplication/row_event.py around line 613. Verify how TABLE_MAP_EVENT handling populates table_map before skipped events are ignored. Done means previously encountered schemas are not queried again for each skipped event.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.