julien-duponchelle / julien-duponchelle/python-mysql-replication

AssertionError after db reconnection

Open
#631 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.4k
Forks
690
PR merge metrics
No merged PRs in 30d

Description

We have:
- a 3 node mariadb 10.6 galera cluster, that uses replication to follow another galera cluster.
- a haproxy node
- an application that connects to the database through haproxy and reads the replication log.

The scenario:
- An update to database config causes the db nodes to be restarted one by one.
- This obviously causes the application to lose connection and reconnect. Because it goes through haproxy it reconnects to a different db node.
- After the reconnection, the AssertionError is raised:

```
AssertionError: Result length not requested length:
Expected=125. Actual=51. Position: 89. Data Length: 140
[...]
File "pymysqlreplication/row_event.py", line 565, in rows
self._fetch_rows()
File "pymysqlreplication/row_event.py", line 560, in _fetch_rows
self.__rows.append(self._fetch_one_row())
File "pymysqlreplication/row_event.py", line 621, in _fetch_one_row
row["values"] = self._read_column_data(self.columns_present_bitmap)
File "pymysqlreplication/row_event.py", line 148, in _read_column_data
values[name] = self.__read_values_name(
File "pymysqlreplication/row_event.py", line 216, in __read_values_name
else self.__read_string(1, column)
File "pymysqlreplication/row_event.py", line 322, in __read_string
string = self.packet.read_length_coded_pascal_string(size)
File "pymysqlreplication/packet.py", line 243, in read_length_coded_pascal_string
return self.read(length)
File "pymysqlreplication/packet.py", line 144, in read
return self.packet.read(size)
File "pymysql/protocol.py", line 73, in read
raise AssertionError(error)
```

I have 12 samples right now. There are different values in "Expected=???. Actual=???. Position: ???. Data Length: ???" and in few cases instead of
```
File "pymysqlreplication/row_event.py", line 216, in __read_values_name
else self.__read_string(1, column)
```
there is
```
File "pymysqlreplication/row_event.py", line 223, in __read_values_name
return self.__read_string(column.length_size, column)
```

This might be a duplicate of #610 .

Unfortunately for now I don't have more data and I won't be restarting production db nodes just for this.

Contributor guide

Open the contributing guide

Research direction

Start with the stack-trace paths in pymysqlreplication/row_event.py and pymysqlreplication/packet.py, then compare the behavior with issue #610. Investigate the assertion after a connection moves to another MariaDB node; done means the post-reconnection parsing failure is reproducible and its resolution is verified against the reported cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
mariadb, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.