Error serializing dataframes with timestamps
- Dominant language
- Python
- Stars
- 521
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 35
Description
Discovered "error serializing datetime" (or timestamps or others), described below, when serializing a complex class with several datetime and other parameters, and several dataframes. Identified that the issue seems to be in how pandas df's are (de)serialized - the "to_dict" function is called (see [here](https://github.com/holoviz/param/blob/7d702adfb4ace592a9550a25e8ca49522813fef4/param/__init__.py#L1566-L1577)) instead of panda's own json functions... in a df with timestamps those are simply dumped out in the resulting dict and that is attempted to be serialized, which fails as the timestamps aren't serializable without handling.
I found that by monkey patching the functions to use pandas own to_json/read_json functions I'm able to get the desired functionality. Also, by using the "table" [orient](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html#pandas-dataframe-to-json) option we get the schema in the output, allowing better serde roundtrip.
If the code below looks OK I'll be happy to submit a PR to add
Versions:
pandas: 1.4.0
json: 2.0.9
param: 1.12.0
python 3.10
osx
Description of expected behavior and the observed behavior:
Serialize and deserialize pandas dataframes with timestamps
Complete, minimal, self-contained example code that reproduces the issue
```
import datetime as dt
import json
import pandas as pd
import param
class TestClass(param.Parameterized):
df_param = param.DataFrame()
df = pd.DataFrame({'a': [dt.datetime(2000, 1, 1), dt.datetime(2000, 1, 1), dt.datetime(2000, 1, 1)],
'b': [1, 2, 3],
'c': [1.0, 2.0, 3.0]})
test = TestClass(df_param=df)
test_serde_dict = TestClass.param.deserialize_parameters(test.param.serialize_parameters())
```
Stack traceback and/or browser JavaScript console output
```
Traceback (most recent call last):
File "/Users/carlsoer/opt/anaconda3/envs/covid-model/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3251, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 15, in
test_serde_dict = TestClass.param.deserialize_parameters(test.param.serialize_parameters())
File "/Users/carlsoer/opt/anaconda3/envs/covid-model/lib/python3.10/site-packages/param/parameterized.py", line 2087, in serialize_parameters
return serializer.serialize_parameters(self_or_cls, subset=subset)
File "/Users/carlsoer/opt/anaconda3/envs/covid-model/lib/python3.10/site-packages/param/serializer.py", line 104, in serialize_parameters
return cls.dumps(components)
File "/Users/carlsoer/opt/anaconda3/envs/covid-model/lib/python3.10/site-packages/param/serializer.py", line 81, in dumps
return json.dumps(obj)
File "/Users/carlsoer/opt/anaconda3/envs/covid-model/lib/python3.10/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/Users/carlsoer/opt/anaconda3/envs/covid-model/lib/python3.10/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/Users/carlsoer/opt/anaconda3/envs/covid-model/lib/python3.10/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/Users/carlsoer/opt/anaconda3/envs/covid-model/lib/python3.10/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Timestamp is not JSON serializable
```
Example fix:
```
def df_serialize(cls, value):
return json.loads(value.to_json(orient='table'))
def df_deserialize(cls, value):
import pandas as pd
return pd.read_json(json.dumps(value), orient='table')
param.DataFrame.serialize = df_serialize
param.DataFrame.deserialize = df_deserialize
test_serde_dict = TestClass.param.deserialize_parameters(test.param.serialize_parameters())
test_serde = TestClass(**test_serde_dict)
print(test_serde.__repr__())
print(test.__repr__())
```
output:
```
TestClass(df_param= a b c
0 2000-01-01 1 1.0
1 2000-01-01 2 2.0
2 2000-01-01 3 3.0, name='TestClass00002')
TestClass(df_param= a b c
0 2000-01-01 1 1.0
1 2000-01-01 2 2.0
2 2000-01-01 3 3.0, name='TestClass00002')
```
Contributor guide
Assessment
This issue has not been assessed yet.