google / google/meridian

Transforming Old Pickle Files into Protobuf

Open
#1,403 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.5k
Forks
294
Avg merge
1d 21h
Merged PRs (30d)
47

Description

Hi Meridian Team,

Given the deprecation of the pickle method in the latest release, how do you recommend transforming old pickle files into the new Protobuf format? This is an issue as we could have multiple previous saved pickle files for each past month and might need to reload a previous model.

Also, when I try to actually save a model I get this warning serialising the prior:
`---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
[/tmp/ipython-input-1787322961.py](https://localhost:8080/#) in ()
1 file_path = '/content/drive/MyDrive/acorns_m11_summary_ds.binpb'
----> 2 meridian_serde.save_meridian(mmm, file_path)

5 frames
[/usr/local/lib/python3.12/dist-packages/schema/serde/meridian_serde.py](https://localhost:8080/#) in save_meridian(mmm, file_path, distribution_function_registry, eda_function_registry)
355 with _file_open(file_path, 'wb') as f:
356 # Creates an MmmKernel.
--> 357 serialized_kernel = MeridianSerde().serialize(
358 mmm,
359 distribution_function_registry=distribution_function_registry,

[/usr/local/lib/python3.12/dist-packages/schema/serde/meridian_serde.py](https://localhost:8080/#) in serialize(self, obj, model_id, meridian_version, include_convergence_info, distribution_function_registry, eda_function_registry)
112 else function_registry_utils.FunctionRegistry()
113 )
--> 114 meridian_model_proto = self._make_meridian_model_proto(
115 mmm=obj,
116 model_id=model_id,

[/usr/local/lib/python3.12/dist-packages/schema/serde/meridian_serde.py](https://localhost:8080/#) in _make_meridian_model_proto(self, mmm, model_id, meridian_version, distribution_function_registry, eda_function_registry, include_convergence_info)
162 prior_tfp_distributions=distribution.DistributionSerde(
163 distribution_function_registry
--> 164 ).serialize(mmm.model_spec.prior),
165 inference_data=inference_data.InferenceDataSerde().serialize(
166 mmm.inference_data

[/usr/local/lib/python3.12/dist-packages/schema/serde/distribution.py](https://localhost:8080/#) in serialize(self, obj)
72 continue
73 getattr(proto, param).CopyFrom(
---> 74 self._to_distribution_proto(getattr(obj, param))
75 )
76 proto.function_registry.update(self.function_registry.hashed_registry)

[/usr/local/lib/python3.12/dist-packages/schema/serde/distribution.py](https://localhost:8080/#) in _to_distribution_proto(self, dist)
161 distribution_type=dist_name,
162 parameters={
--> 163 name: self._to_parameter_value_proto(name, value, dist_class)
164 for name, value in dist.parameters.items()
165 },

[/usr/local/lib/python3.12/dist-packages/schema/serde/distribution.py](https://localhost:8080/#) in _to_parameter_value_proto(self, param_name, value, dist)
260
261 # Handle unsupported types.
--> 262 raise TypeError(f"Unsupported type: {type(value)}, {value}")
263
264 def _from_distribution_proto(

TypeError: Unsupported type: , [ 1.6 1.6 4.5 1.6 4.5 4. 6. 1.6 1.6 5. 1. 18. ]`

Contributor guide

Open the contributing guide

Research direction

Start with schema/serde/meridian_serde.py at save_meridian and schema/serde/distribution.py at distribution parameter serialization, as shown in the traceback. Reproduce the Protobuf save failure with the reported numpy.ndarray prior, then investigate how legacy pickle files are handled. Done means the migration path and reported save error have a documented or verified resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.