julien-duponchelle / julien-duponchelle/python-mysql-replication

Replaying binlog with dropped columns causes unhandled exception.

Open
#118 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.4k
Forks
690
PR merge metrics
No merged PRs in 30d

Description

Steps to reproduce:
1. create a table with few columns
2. generate some row events
3. drop a column from that table
4. generate some more row events
5. start `pymysqlreplication` replaying the binlog from the position prior to dropping the column

ER:
Working fine.

AR:
Unhandled error:

```
File "/usr/local/lib/python2.7/dist-packages/pymysqlreplication/binlogstream.py", line 262, in fetchone
self.__freeze_schema)
File "/usr/local/lib/python2.7/dist-packages/pymysqlreplication/packet.py", line 98, in __init__
freeze_schema = freeze_schema)
File "/usr/local/lib/python2.7/dist-packages/pymysqlreplication/row_event.py", line 550, in __init__
column_schema = self.column_schemas[i]
IndexError: list index out of range
```

This is partially solved by 4c48538168f4cd3239563393a29b542cc6ffcf4b, but that doesn't solve the deeper issue at hand which is how the schema for tables is obtained. Schema is always obtained from the current version of `information_schema` no matter how far in the past the RowEvent processed is.

Contributor guide

Open the contributing guide

Research direction

Reproduce the dropped-column scenario, then trace binlogstream.py:262 into packet.py:98 and row_event.py:550, comparing the behavior with commit 4c48538168f4cd3239563393a29b542cc6ffcf4b. Investigate how the replay obtains table schema from information_schema for older RowEvents. Done means replaying events across the column drop without an unhandled IndexError and using schema appropriate to the processed history.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.