microsoft / microsoft/SynapseML
Using TrainClassifier with class weights (including ClassBalancer)
@imatiach-msft is already working on this.
Since Apr 25, 2019.
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
Hi,
I am having a problem with using the TrainClassifier when the label's classes have weights (including instance weights generated by the MMLSpark Class Balancer class).
I am using Pyspark 3 on Spark 2.2.0.
The non-MMLSpark way of creating a classifier model with weights, for a table with labels defined by a variable label_col, feature column defined by a variable col, and weight column weight, would be something like:
`LogisticRegression().setWeightCol("weight").setLabelCol(label_col).setFeaturesCol(col)
`
I tried different combinations to do it using MMLSpark:
**#1:**
```
base_model = LogisticRegression().setWeightCol('weight').setFeaturesCol(col).setLabelCol(label_col)
TrainClassifier(model=base_model)
```
will lead to the following error when training the model:
```
Py4JJavaError: An error occurred while calling o17763.fit.
: java.util.NoSuchElementException: Failed to find a default value for labelCol
```
presumably because the TrainClassifier constructor requires setLabelCol to be set explicitly.
**#2:**
```
base_model = LogisticRegression().setWeightCol('weight').setFeaturesCol(col)
TrainClassifier(model=base_model).setLabelCol(label_col)
```
will lead to a model with 2 coefficients for a single feature and 1 intercept. Presumably, because the weight column was used as a feature, and the feature column set by the base method's "setFeaturesCol" was ignored by TrainClassifier.
**#3:**
```
base_model = LogisticRegression().setWeightCol('weight')
TrainClassifier(model=base_model).setLabelCol(label_col).setFeaturesCol(col)
```
will lead to the following error when training the model:
```
Py4JJavaError: An error occurred while calling o17925.fit.
: org.apache.spark.sql.AnalysisException: Reference '__' is ambiguous, could be: __#13901, __#32151.;
```
presumably because the TrainClassifier constructor changes the names of the features, and the setFeaturesCol doesn't work properly it seems.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.