microsoft / microsoft/SynapseML

TabularLime errors

Open
#570 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

I'm getting the following errors when using TabularLime:
"Output column features already exists."

Based off the [tests ](https://github.com/Azure/mmlspark/blob/99d267a407c07301dcada42f788ef7b92ef15aa5/src/image-featurizer/src/test/scala/LIMESuite.scala) for TabularLime, I should be calling a "fit" on TabularLime and then a "transform".

I'm on Spark 2.3.0 (Cloudera). The following shows my code to replicate this error:

```python
from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier

spark = SparkSession\
.builder\
.appName("pyspark")\
.getOrCreate()

from mmlspark import *
from mmlspark import FluentAPI

def generateDataFrame(spark):
data=[
{'x1':3, 'x2':4, 'y':1},
{'x1':2, 'x2':1, 'y':0},
{'x1':2, 'x2':2, 'y':0},
{'x1':3, 'x2':5, 'y':1}
]
return spark.createDataFrame(data)

vectorAssembler = VectorAssembler()\
.setInputCols(["x1", "x2"])\
.setOutputCol("features")

model=RandomForestClassifier(featuresCol="features", labelCol="y")

pipeline = Pipeline(stages=[
vectorAssembler, model
])

data=generateDataFrame(spark)

fitmodel=pipeline.fit(data)

lime = TabularLIME()\
.setModel(fitmodel)\
.setPredictionCol("prediction")\
.setOutputCol("weights")\
.setInputCol("features")

result = lime.fit(fitmodel.transform(data))

#This is where the error occurs
result.transform(fitmodel.transform(data))

spark.stop()
```

Contributor guide

Open the contributing guide

Research direction

Start with the TabularLIME fit/transform flow in the linked LIMESuite.scala test and reproduce the reported error using the provided PySpark example on Spark 2.3.0. Check how the existing features and prediction columns are handled across fit and transform. Done means the example completes without the "Output column features already exists" error and the relevant test covers the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
machine-learning, python, scala
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.