python SimpleSparkSerializer support protobuf as serialize format
Open
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 315
- PR merge metrics
- No merged PRs in 30d
Description
[SimpleSparkSerializer](https://github.com/combust/mleap/blob/087e8f04f79d08bad5e37600c4668dfb0328013f/python/mleap/pyspark/spark_support.py#L36) use the default `json` format.
But for model like Word2Vec, size can be huge and json decode with current design will cause heap OOM.
To ease this problem, protobuf format will provide better model size and read/write performance with less memory consumption.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.