julien-duponchelle / julien-duponchelle/python-mysql-replication
AssertionError after db reconnection
- Dominant language
- Python
- Stars
- 2.4k
- Forks
- 690
- PR merge metrics
- No merged PRs in 30d
Description
We have:
- a 3 node mariadb 10.6 galera cluster, that uses replication to follow another galera cluster.
- a haproxy node
- an application that connects to the database through haproxy and reads the replication log.
The scenario:
- An update to database config causes the db nodes to be restarted one by one.
- This obviously causes the application to lose connection and reconnect. Because it goes through haproxy it reconnects to a different db node.
- After the reconnection, the AssertionError is raised:
```
AssertionError: Result length not requested length:
Expected=125. Actual=51. Position: 89. Data Length: 140
[...]
File "pymysqlreplication/row_event.py", line 565, in rows
self._fetch_rows()
File "pymysqlreplication/row_event.py", line 560, in _fetch_rows
self.__rows.append(self._fetch_one_row())
File "pymysqlreplication/row_event.py", line 621, in _fetch_one_row
row["values"] = self._read_column_data(self.columns_present_bitmap)
File "pymysqlreplication/row_event.py", line 148, in _read_column_data
values[name] = self.__read_values_name(
File "pymysqlreplication/row_event.py", line 216, in __read_values_name
else self.__read_string(1, column)
File "pymysqlreplication/row_event.py", line 322, in __read_string
string = self.packet.read_length_coded_pascal_string(size)
File "pymysqlreplication/packet.py", line 243, in read_length_coded_pascal_string
return self.read(length)
File "pymysqlreplication/packet.py", line 144, in read
return self.packet.read(size)
File "pymysql/protocol.py", line 73, in read
raise AssertionError(error)
```
I have 12 samples right now. There are different values in "Expected=???. Actual=???. Position: ???. Data Length: ???" and in few cases instead of
```
File "pymysqlreplication/row_event.py", line 216, in __read_values_name
else self.__read_string(1, column)
```
there is
```
File "pymysqlreplication/row_event.py", line 223, in __read_values_name
return self.__read_string(column.length_size, column)
```
This might be a duplicate of #610 .
Unfortunately for now I don't have more data and I won't be restarting production db nodes just for this.
Contributor guide
Research direction
Start with the stack-trace paths in pymysqlreplication/row_event.py and pymysqlreplication/packet.py, then compare the behavior with issue #610. Investigate the assertion after a connection moves to another MariaDB node; done means the post-reconnection parsing failure is reproducible and its resolution is verified against the reported cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mariadb, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100