hazelcast / hazelcast/hazelcast-python-client

PythonObjectSerializer raise 'utf-8' codec can't decode byte

Open
#639 2 comments 0 reactions 0 assignees View on GitHub
Source: Community
Dominant language
Python
Stars
116
Forks
78
Avg merge
10d 22h
Merged PRs (30d)
1

Description

Good time of the day team

When I tried to cache API responses, it failed as sometimes I have UnicodeDecodeError.

Looking deeper, I found that this is happening because of this code `out.write_string(cPickle.dumps(obj, 0).decode("utf-8"))` in `PythonObjectSerializer`
Full Code:
```python
class PythonObjectSerializer(BaseSerializer):
def read(self, inp):
str = inp.read_string().encode()
return cPickle.loads(str)

def write(self, out, obj):
out.write_string(cPickle.dumps(obj, 0).decode("utf-8"))

def get_type_id(self):
return PYTHON_TYPE_PICKLE
```

Issue example:
```console
>>> import pickle

>>> pickle.dumps("\u00e4").decode("utf-8")

Traceback (most recent call last):

File "", line 1, in

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
````

As a workaround, I created such a custom serializer:
```
class HazelcastJsonSerializer(StreamSerializer):

def read(self, inp):
return json.loads(inp.read_string())

def write(self, out, obj):
out.write_string(json.dumps(obj))

def get_type_id(self):

```

Is there any better solution?

python version: 3.6

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.