Spark OneHotEncoderEstimator without input/output cols can be serialized but not de-serialized
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 315
- PR merge metrics
- No merged PRs in 30d
Description
I'm attempting to serialize a Spark `pipelineModel` to an Mleap bundle and then de-serialize it to mleap. I noticed that if the input and output columns are not set for a `OneHotEncoderEstimator`, it is serialized successfully with empty lists for in/out shapes. When de-serializing to Mleap I get this error, presumably because the shapes are empty.
```
java.util.NoSuchElementException: head of empty list
at scala.collection.immutable.Nil$.head(List.scala:431)
at scala.collection.immutable.Nil$.head(List.scala:428)
at ml.combust.mleap.runtime.frame.SimpleTransformer$class.$init$(Transformer.scala:108)
at ml.combust.mleap.runtime.transformer.feature.OneHotEncoder.(OneHotEncoder.scala:18)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at ml.combust.mleap.bundle.ops.feature.OneHotEncoderOp.load(OneHotEncoderOp.scala:57)
at ml.combust.mleap.bundle.ops.feature.OneHotEncoderOp.load(OneHotEncoderOp.scala:17)
at ml.combust.bundle.serializer.NodeSerializer$$anonfun$read$2$$anonfun$apply$3$$anonfun$apply$4.apply(NodeSerializer.scala:106)
at scala.util.Try$.apply(Try.scala:192)
at ml.combust.bundle.serializer.NodeSerializer$$anonfun$read$2$$anonfun$apply$3.apply(NodeSerializer.scala:104)
at ml.combust.bundle.serializer.NodeSerializer$$anonfun$read$2$$anonfun$apply$3.apply(NodeSerializer.scala:102)
at scala.util.Success.flatMap(Try.scala:231)
at ml.combust.bundle.serializer.NodeSerializer$$anonfun$read$2.apply(NodeSerializer.scala:102)
at ml.combust.bundle.serializer.NodeSerializer$$anonfun$read$2.apply(NodeSerializer.scala:101)
at scala.util.Success.flatMap(Try.scala:231)
at ml.combust.bundle.serializer.NodeSerializer.read(NodeSerializer.scala:100)
at ml.combust.bundle.serializer.GraphSerializer$$anonfun$readNode$2.apply(GraphSerializer.scala:57)
at ml.combust.bundle.serializer.GraphSerializer$$anonfun$readNode$2.apply(GraphSerializer.scala:57)
at scala.util.Success.flatMap(Try.scala:231)
at ml.combust.bundle.serializer.GraphSerializer.readNode(GraphSerializer.scala:56)
at ml.combust.bundle.serializer.GraphSerializer$$anonfun$read$1.apply(GraphSerializer.scala:44)
at ml.combust.bundle.serializer.GraphSerializer$$anonfun$read$1.apply(GraphSerializer.scala:44)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:296)
at ml.combust.bundle.serializer.GraphSerializer.read(GraphSerializer.scala:44)
at ml.combust.mleap.bundle.ops.PipelineOp$$anon$1.load(PipelineOp.scala:28)
at ml.combust.mleap.bundle.ops.PipelineOp$$anon$1.load(PipelineOp.scala:15)
at ml.combust.bundle.serializer.ModelSerializer$$anonfun$readWithModel$2.apply(ModelSerializer.scala:106)
at ml.combust.bundle.serializer.ModelSerializer$$anonfun$readWithModel$2.apply(ModelSerializer.scala:104)
at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Success.map(Try.scala:237)
at ml.combust.bundle.serializer.ModelSerializer.readWithModel(ModelSerializer.scala:103)
at ml.combust.bundle.serializer.NodeSerializer$$anonfun$read$2.apply(NodeSerializer.scala:102)
at ml.combust.bundle.serializer.NodeSerializer$$anonfun$read$2.apply(NodeSerializer.scala:101)
at scala.util.Success.flatMap(Try.scala:231)
at ml.combust.bundle.serializer.NodeSerializer.read(NodeSerializer.scala:100)
at ml.combust.bundle.serializer.BundleSerializer$$anonfun$read$2.apply(BundleSerializer.scala:55)
at ml.combust.bundle.serializer.BundleSerializer$$anonfun$read$2.apply(BundleSerializer.scala:49)
at scala.util.Success.flatMap(Try.scala:231)
at ml.combust.bundle.serializer.BundleSerializer.read(BundleSerializer.scala:49)
at ml.combust.bundle.BundleFile.load(BundleFile.scala:123)
at ml.combust.mleap.runtime.MleapSupport$MleapBundleFileOps.loadMleapBundle(MleapSupport.scala:25)
at $anonfun$2.apply(:19)
at $anonfun$2.apply(:18)
at resource.AbstractManagedResource$$anonfun$5.apply(AbstractManagedResource.scala:88)
at scala.util.control.Exception$Catch$$anonfun$either$1.apply(Exception.scala:125)
at scala.util.control.Exception$Catch$$anonfun$either$1.apply(Exception.scala:125)
at scala.util.control.Exception$Catch.apply(Exception.scala:103)
at scala.util.control.Exception$Catch.either(Exception.scala:125)
at resource.AbstractManagedResource.acquireFor(AbstractManagedResource.scala:88)
at resource.DeferredExtractableManagedResource.either(AbstractManagedResource.scala:29)
at resource.DeferredExtractableManagedResource.opt(AbstractManagedResource.scala:31)
... 42 elided
```
The serialization code for the model looks like this:
```
val bundleFileSystem = new HadoopBundleFileSystem(modelWriteHadoopFs, Seq("s3a", "hdfs"))
val tmpDir = Files.createTempDirectory("bundle")
val tmp = Paths.get(tmpDir.toString, "tmp.zip")
implicit val context: SparkBundleContext = SparkBundleContext().withDataset(transformedDf)
(for (bf <- resource.managed(BundleFile(tmp.toFile))) yield {
model.writeBundle.save(bf)(context)
}).tried.map {
_ => bundleFileSystem.save(new URI(s3Path ++ "/model.zip"), tmp.toFile)
}
```
The de-serialization looks like this:
```
new URI("jar:file:/data/model.zip").loadMleapBundle().get.root
```
It would make more sense to have consistent behavior across serialization and de-serialization, in one of the following ways:
1) Serialization throws error when `OneHotEncoderEstimator` has no defined in/out cols
2) `OneHotEncoderEstimator` without in/out cols is not serialized to bundle
3) `OneHotEncoderEstimator` without in/out cols is de-serialized without error
If any of these sounds like a better alternative to the current behavior, I would be happy to contribute a fix.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.