julien-duponchelle / julien-duponchelle/python-mysql-replication

Unable to decode with 'utf-8'

Open
#313 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.4k
Forks
690
PR merge metrics
No merged PRs in 30d

Description

Hello,

While decoding the `utf8mb4` from MYSQL to `utf-8`. This is what I got this

```
streamer.run()File "/usr/src/app/lib/streamer.py", line 139, in runfor row in binlog_event.rows:
File "/usr/local/lib/python3.6/site-packages/pymysqlreplication/row_event.py", line 429, in rowsself._fetch_rows()
File "/usr/local/lib/python3.6/site-packages/pymysqlreplication/row_event.py", line 424, in _fetch_rowsself.__rows.append(self._fetch_one_row())
File "/usr/local/lib/python3.6/site-packages/pymysqlreplication/row_event.py", line 449, in _fetch_one_rowrow["values"] = self._read_column_data(self.columns_present_bitmap)
File "/usr/local/lib/python3.6/site-packages/pymysqlreplication/row_event.py", line 136, in _read_column_datavalues[name] = self.__read_string(column.length_size, column)
File "/usr/local/lib/python3.6/site-packages/pymysqlreplication/row_event.py", line 220, in __read_stringstring = string.decode(charset_to_encoding(column.character_set_name))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte
```

I was trying to decode this `\xed\xa0\x8d` invalid sequence of `UTF-8`

Schema:
`"CREATE TABLE test (test LONGTEXT COLLATE utf8mb4_unicode_ci) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;`

Tentative Solution:

Wrapping `string.decode(charset_to_encoding(column.character_set_name)` in try/catch for now working for me. I believe it's because of UTF-8 encoding having an invalid byte sequence or the character has a byte sequence greater than `U+10FFFD`.

Thanks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.