microsoft / microsoft/SynapseML
TabularLime errors
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
I'm getting the following errors when using TabularLime:
"Output column features already exists."
Based off the [tests ](https://github.com/Azure/mmlspark/blob/99d267a407c07301dcada42f788ef7b92ef15aa5/src/image-featurizer/src/test/scala/LIMESuite.scala) for TabularLime, I should be calling a "fit" on TabularLime and then a "transform".
I'm on Spark 2.3.0 (Cloudera). The following shows my code to replicate this error:
```python
from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
spark = SparkSession\
.builder\
.appName("pyspark")\
.getOrCreate()
from mmlspark import *
from mmlspark import FluentAPI
def generateDataFrame(spark):
data=[
{'x1':3, 'x2':4, 'y':1},
{'x1':2, 'x2':1, 'y':0},
{'x1':2, 'x2':2, 'y':0},
{'x1':3, 'x2':5, 'y':1}
]
return spark.createDataFrame(data)
vectorAssembler = VectorAssembler()\
.setInputCols(["x1", "x2"])\
.setOutputCol("features")
model=RandomForestClassifier(featuresCol="features", labelCol="y")
pipeline = Pipeline(stages=[
vectorAssembler, model
])
data=generateDataFrame(spark)
fitmodel=pipeline.fit(data)
lime = TabularLIME()\
.setModel(fitmodel)\
.setPredictionCol("prediction")\
.setOutputCol("weights")\
.setInputCol("features")
result = lime.fit(fitmodel.transform(data))
#This is where the error occurs
result.transform(fitmodel.transform(data))
spark.stop()
```
Contributor guide
Research direction
Start with the TabularLIME fit/transform flow in the linked LIMESuite.scala test and reproduce the reported error using the provided PySpark example on Spark 2.3.0. Check how the existing features and prediction columns are handled across fit and transform. Done means the example completes without the "Output column features already exists" error and the relevant test covers the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- machine-learning, python, scala
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100