combust / combust/mleap

python SimpleSparkSerializer support protobuf as serialize format

Open
#817 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.5k
Forks
315
PR merge metrics
No merged PRs in 30d

Description

[SimpleSparkSerializer](https://github.com/combust/mleap/blob/087e8f04f79d08bad5e37600c4668dfb0328013f/python/mleap/pyspark/spark_support.py#L36) use the default `json` format.
But for model like Word2Vec, size can be huge and json decode with current design will cause heap OOM.
To ease this problem, protobuf format will provide better model size and read/write performance with less memory consumption.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.