Transforming Old Pickle Files into Protobuf
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 294
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 47
Description
Hi Meridian Team,
Given the deprecation of the pickle method in the latest release, how do you recommend transforming old pickle files into the new Protobuf format? This is an issue as we could have multiple previous saved pickle files for each past month and might need to reload a previous model.
Also, when I try to actually save a model I get this warning serialising the prior:
`---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
[/tmp/ipython-input-1787322961.py](https://localhost:8080/#) in ()
1 file_path = '/content/drive/MyDrive/acorns_m11_summary_ds.binpb'
----> 2 meridian_serde.save_meridian(mmm, file_path)
5 frames
[/usr/local/lib/python3.12/dist-packages/schema/serde/meridian_serde.py](https://localhost:8080/#) in save_meridian(mmm, file_path, distribution_function_registry, eda_function_registry)
355 with _file_open(file_path, 'wb') as f:
356 # Creates an MmmKernel.
--> 357 serialized_kernel = MeridianSerde().serialize(
358 mmm,
359 distribution_function_registry=distribution_function_registry,
[/usr/local/lib/python3.12/dist-packages/schema/serde/meridian_serde.py](https://localhost:8080/#) in serialize(self, obj, model_id, meridian_version, include_convergence_info, distribution_function_registry, eda_function_registry)
112 else function_registry_utils.FunctionRegistry()
113 )
--> 114 meridian_model_proto = self._make_meridian_model_proto(
115 mmm=obj,
116 model_id=model_id,
[/usr/local/lib/python3.12/dist-packages/schema/serde/meridian_serde.py](https://localhost:8080/#) in _make_meridian_model_proto(self, mmm, model_id, meridian_version, distribution_function_registry, eda_function_registry, include_convergence_info)
162 prior_tfp_distributions=distribution.DistributionSerde(
163 distribution_function_registry
--> 164 ).serialize(mmm.model_spec.prior),
165 inference_data=inference_data.InferenceDataSerde().serialize(
166 mmm.inference_data
[/usr/local/lib/python3.12/dist-packages/schema/serde/distribution.py](https://localhost:8080/#) in serialize(self, obj)
72 continue
73 getattr(proto, param).CopyFrom(
---> 74 self._to_distribution_proto(getattr(obj, param))
75 )
76 proto.function_registry.update(self.function_registry.hashed_registry)
[/usr/local/lib/python3.12/dist-packages/schema/serde/distribution.py](https://localhost:8080/#) in _to_distribution_proto(self, dist)
161 distribution_type=dist_name,
162 parameters={
--> 163 name: self._to_parameter_value_proto(name, value, dist_class)
164 for name, value in dist.parameters.items()
165 },
[/usr/local/lib/python3.12/dist-packages/schema/serde/distribution.py](https://localhost:8080/#) in _to_parameter_value_proto(self, param_name, value, dist)
260
261 # Handle unsupported types.
--> 262 raise TypeError(f"Unsupported type: {type(value)}, {value}")
263
264 def _from_distribution_proto(
TypeError: Unsupported type: , [ 1.6 1.6 4.5 1.6 4.5 4. 6. 1.6 1.6 5. 1. 18. ]`
Contributor guide
Research direction
Start with schema/serde/meridian_serde.py at save_meridian and schema/serde/distribution.py at distribution parameter serialization, as shown in the traceback. Reproduce the Protobuf save failure with the reported numpy.ndarray prior, then investigate how legacy pickle files are handled. Done means the migration path and reported save error have a documented or verified resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100