Input schema of a StringIndexed Column is always string
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 315
- PR merge metrics
- No merged PRs in 30d
Description
I have a dataset with a numerical column `rank` with values ranging from 1 to 5. When this dataset is read in spark with `inferSchema = true`, the dataType of the column is inferred as `int` by spark. A model is built by applying `StringIndexer` on the `rank` column as one of the stages and the `org.apache.spark.ml.PipelineModel` is exported as an mleap bundle. When we read the `ml.combust.mleap.runtime.frame.Transformer` from the mleap bundle and observe the `transformer.inputSchema`, it returns the `dataType` of the `rank` column as `String` instead of an `int`.
Looks like this is because of
https://github.com/combust/mleap/blob/master/mleap-core/src/main/scala/ml/combust/mleap/core/feature/StringIndexerModel.scala#L50
According to spark docs for [StringIndexer](https://spark.apache.org/docs/2.1.0/ml-features.html#stringindexer),
> If the input column is numeric, we cast it to string and index the string values.
so, an input column for a string indexer stage can be a numeric or string type.
Is there a way to pass the information of the actual datatype of the column to the transformer, so that `transformer.inputSchema` can return same instead of `String` for all the cases?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.