julien-duponchelle / julien-duponchelle/python-mysql-replication
Critical Bug - Usage of skip_to_timestamp option causes querying the information schema huge number of times
- Dominant language
- Python
- Stars
- 2.4k
- Forks
- 690
- PR merge metrics
- No merged PRs in 30d
Description
### Overview
We are using the library to periodically sync the updated rows to another database. We are using the `skip_to_timestamp` option to skip the binary logs that were already synced in the previous cycle. But if we use this, we see that the library is executing [this](https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/binlogstream.py#L606-L614) query too many times.
### Bug description
According to the logic written in *row_event.py* ([line 613](https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/row_event.py#L613)), it seems the schema is fetched for every TABLE_MAP_EVENT if it is not already present in the *table_map*. But in binlogstream.py ([lines 551 to 557](https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/binlogstream.py#L551-L557)), the result of the fetched schema is ignored if the event timestamp is lesser than the `skip_to_timestamp` option. So, the schema will be fetched again in the next TABLE_MAP_EVENT as it was not populated in the *table_map* previously.
### Resolution
Populating the *table_map* first before continuing the loop in binlogstream.py ([lines 551 to 557](https://github.com/julien-duponchelle/python-mysql-replication/blob/main/pymysqlreplication/binlogstream.py#L551-L557)) should fix the issue.
Contributor guide
Research direction
Start by reading the skip_to_timestamp loop in pymysqlreplication/binlogstream.py around lines 551-557, then follow the table-map schema lookup in pymysqlreplication/row_event.py around line 613. Verify how TABLE_MAP_EVENT handling populates table_map before skipped events are ignored. Done means previously encountered schemas are not queried again for each skipped event.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100